…
The package miRNAselector has a lot of requirements that are nessesary to run all the experiments. The script below will allow to install most of them. It is highly recommended to install those packages using the code below.
What should be done? - If you have Nvidia GPU - install CUDA and set gpu=T in the setup script below.
readLines("setup.R") %>% paste0(collapse = "\n") %>% cat
#> install.packages("devtools",repos = "http://cran.r-project.org")
#> install.packages("BiocManager",repos = "http://cran.r-project.org")
#> suppressMessages(library(BiocManager))
#> BiocManager::install(c("reticulate","devtools","plyr","dplyr","edgeR","epiDisplay","rsq","MASS","Biocomb","caret","dplyr",
#> "pROC","ggplot2","DMwR", "doParallel", "Boruta", "spFSR", "varSelRF", "stringr", "psych", "C50", "randomForest",
#> "foreach","data.table", "ROSE", "deepnet", "gridExtra", "stargazer","gplots","My.stepwise","snow",
#> "calibrate", "ggrepel", "networkD3", "VennDiagram","RSNNS", "kernlab", "car", "PairedData",
#> "profileR","classInt","kernlab","xgboost", "keras", "tidyverse", "cutpointr","tibble","tidyr", "rpart", "party", "mgcv", "GDCRNATools",
#> "imputeMissings", "visdat", "naniar", "stringr", "doSNOW", "R.utils"))
#> remotes::install_github("STATWORX/bounceR", force = T)
#> #remotes::install_github("rstudio/reticulate")
#> remotes::install_github("vqv/ggbiplot", force = T)
#> suppressMessages(library(keras))
#> install_keras()
#>
#>
#> remotes::install_github("kstawiski/miRNAselector", force = T) # Install our package.This code does not cover the installation of mxnet, which can be used for benchmarking of the selected miRNA sets.
In this showcase we will use TCGA-derived data.
miRNAselecter has built-it functions for downloading of miRNA-seq data from TCGA. By default all projects are downloaded and processed using 2 functions as shown below.
ks.download_tissue_miRNA_data_from_TCGA()
ks.process_tissue_miRNA_TCGA(remove_miRNAs_with_null_var = T)
Both of those function produce 2 files: tissue_miRNA_counts.csv and tissue_miRNA_logtpm.csv. First of those files contains metadata and raw counts as declared in TCGA. The second is are log-transformed transcripts-per-million (TPM) counts. Let’s load counts files and see what sample do we have.
suppressWarnings(suppressMessages(library(data.table)))
suppressWarnings(suppressMessages(library(knitr)))
orginal_TCGA_data = fread("tissue_miRNA_counts.csv.gz")
orginal_TCGA_data[orginal_TCGA_data == ""] = NA
kable(table(orginal_TCGA_data$primary_site, orginal_TCGA_data$sample_type))| PrimaryTumor | SolidTissueNormal | |
|---|---|---|
| Adrenal gland | 80 | 0 |
| Bladder | 409 | 19 |
| Brain | 512 | 5 |
| Breast | 1078 | 104 |
| Bronchus and lung | 991 | 91 |
| Cervix uteri | 307 | 3 |
| Corpus uteri|Stomach|Other and unspecified parts of tongue|Meninges|Other and unspecified male genital organs|Colon|Connective, subcutaneous and other soft tissues|Bones, joints and articular cartilage of limbs|Ovary|Retroperitoneum and peritoneum|Peripheral nerves and autonomic nervous system|Uterus, NOS|Kidney | 259 | 0 |
| Corpus uteri|Uterus, NOS | 538 | 33 |
| Esophagus|Stomach | 184 | 13 |
| Eye and adnexa | 80 | 0 |
| Heart, mediastinum, and pleura|Bronchus and lung | 87 | 0 |
| Heart, mediastinum, and pleura|Other endocrine glands and related structures|Adrenal gland|Connective, subcutaneous and other soft tissues|Other and ill-defined sites|Spinal cord, cranial nerves, and other parts of central nervous system|Retroperitoneum and peritoneum | 179 | 3 |
| Heart, mediastinum, and pleura|Testis|Stomach|Lymph nodes|Bones, joints and articular cartilage of other and unspecified sites|Brain|Thyroid gland|Small intestine|Colon|Connective, subcutaneous and other soft tissues|Other and unspecified major salivary glands|Retroperitoneum and peritoneum|Hematopoietic and reticuloendothelial systems|Breast | 47 | 0 |
| Heart, mediastinum, and pleura|Thymus | 124 | 2 |
| Kidney | 873 | 130 |
| Liver and intrahepatic bile ducts | 372 | 50 |
| Other and ill-defined sites in lip, oral cavity and pharynx|Palate|Other and unspecified parts of tongue|Hypopharynx|Tonsil|Oropharynx|Larynx|Other and unspecified parts of mouth|Gum|Floor of mouth|Bones, joints and articular cartilage of other and unspecified sites|Lip|Base of tongue | 523 | 44 |
| Other and unspecified parts of biliary tract|Gallbladder|Liver and intrahepatic bile ducts | 36 | 9 |
| Ovary | 489 | 0 |
| Pancreas | 178 | 4 |
| Prostate gland | 494 | 52 |
| Rectosigmoid junction|Colon | 444 | 8 |
| Rectosigmoid junction|Unknown|Rectum|Colon|Connective, subcutaneous and other soft tissues | 161 | 3 |
| Skin | 97 | 2 |
| Stomach | 436 | 41 |
| Testis | 150 | 0 |
| Thyroid gland | 506 | 59 |
| Uterus, NOS | 57 | 0 |
Let’s consider a following exemplary problem..
We want to find the set of miRNAs the most specific to pancreatic cancer. We see that there are 178 cases of pancreatic cancer miRNA-seq results and only 4 solid tissue normal cases. However, we have multiple other normal miRNA-seq results that could be incorporated in the analysis. Let’s filter and label the samples of interest.
suppressWarnings(suppressMessages(library(dplyr)))
cancer_cases = filter(orginal_TCGA_data, primary_site == "Pancreas" & sample_type == "PrimaryTumor")
control_cases = filter(orginal_TCGA_data, sample_type == "SolidTissueNormal")The pipeline requires the variable Class to be present in the dataset. This variable has to be present and have only 2 levels: Cancer and Control.
cancer_cases$Class = "Cancer"
control_cases$Class = "Control"
dataset = rbind(cancer_cases, control_cases)
kable(table(dataset$Class), col.names = c("Class", "Number of cases"))| Class | Number of cases |
|---|---|
| Cancer | 178 |
| Control | 675 |
Let’s explore some of the associations between the group.
t.test(dataset$age_at_diagnosis ~ dataset$Class)
#>
#> Welch Two Sample t-test
#>
#> data: dataset$age_at_diagnosis by dataset$Class
#> t = 3.88, df = 360.55, p-value = 0.0001242
#> alternative hypothesis: true difference in means is not equal to 0
#> 95 percent confidence interval:
#> 693.0905 2117.7294
#> sample estimates:
#> mean in group Cancer mean in group Control
#> 23782.23 22376.82
kable(table(dataset$gender.x, dataset$Class))| Cancer | Control | |
|---|---|---|
| female | 80 | 334 |
| male | 98 | 325 |
chisq.test(dataset$gender.x, dataset$Class)
#>
#> Pearson's Chi-squared test with Yates' continuity correction
#>
#> data: dataset$gender.x and dataset$Class
#> X-squared = 1.6241, df = 1, p-value = 0.2025There is the stistically significant difference in age between classess. The gender was not associated with class. Let’s do propensity score matching to balance the sets.
old_dataset = dataset # backup
dataset = dataset[grepl("Adenocarcinomas", dataset$disease_type), ]
match_by = c("age_at_diagnosis", "gender.x")
tempdane = dplyr::select(dataset, match_by)
tempdane$Class = ifelse(dataset$Class == "Cancer", TRUE, FALSE)
suppressMessages(library(mice))
suppressMessages(library(MatchIt))
temp1 = mice(tempdane, m = 1)
#>
#> iter imp variable
#> 1 1 age_at_diagnosis
#> 2 1 age_at_diagnosis
#> 3 1 age_at_diagnosis
#> 4 1 age_at_diagnosis
#> 5 1 age_at_diagnosis
temp2 = temp1$data
temp3 = mice::complete(temp1)
temp3 = temp3[complete.cases(temp3), ]
tempform = ks.create_miRNA_formula(match_by)
mod_match <- matchit(tempform, data = temp3)
newdata = match.data(mod_match)
dataset = dataset[as.numeric(rownames(newdata)), ]
boxplot(dataset$age_at_diagnosis ~ dataset$Class)t.test(dataset$age_at_diagnosis ~ dataset$Class)
#>
#> Welch Two Sample t-test
#>
#> data: dataset$age_at_diagnosis by dataset$Class
#> t = 0.090876, df = 351.63, p-value = 0.9276
#> alternative hypothesis: true difference in means is not equal to 0
#> 95 percent confidence interval:
#> -827.5374 907.7172
#> sample estimates:
#> mean in group Cancer mean in group Control
#> 23782.23 23742.14
kable(table(dataset$gender.x, dataset$Class))| Cancer | Control | |
|---|---|---|
| female | 80 | 79 |
| male | 98 | 99 |
chisq.test(dataset$gender.x, dataset$Class)
#>
#> Pearson's Chi-squared test with Yates' continuity correction
#>
#> data: dataset$gender.x and dataset$Class
#> X-squared = 0, df = 1, p-value = 1
fwrite(dataset, "balanced_dataset.csv.gz")Dataset is a bit better balanced now. In the next steps we will: 1. In order to stay consistent between different datasets we will use ks.correct_miRNA_names() to unify the miRNA names between different versions of miRbase. 2. We will perform standard filtering, log-transormation and TPM-normalization.
dataset = ks.correct_miRNA_names(dataset)
danex = dplyr::select(dataset, starts_with("hsa")) # Create data.frame or matrix with miRNA counts with miRNAs in columns and cases in rows.
metadane = dplyr::select(dataset, -starts_with("hsa")) # Metadata with 'Class' variables.
kable(table(metadane$Class)) # Let's be sure that 'Class' variable is correct and contains only 'Cancer' and 'Control' cases.| Var1 | Freq |
|---|---|
| Cancer | 178 |
| Control | 178 |
ttpm = ks.counts_to_log10tpm(danex, metadane, ids = metadane$sample, filtr = T, filtr_minimalcounts = 100, filtr_howmany = 1/3) # We will leave only the miRNAs which apeared with at least 100 counts in 1/3 of cases.
#>
#> DGEList unfiltered object with TPM was saved as TPM_DGEList.rds.
#> DGEList filtered object with TPM was saved as TPM_DGEList_filtered.rds.
#> (After filtering) miRNAs left: 166 | filtered out: 2418.
#> Returned data are log10(TPM).After application of filter there are still 166 miRNAs left. The filter was applied to ensure our potential diagnostic test may not relay strictly on miRNA-seq data, but the creation of much cheaper qPCR test will be possible.
In the next step we will devide the dataset into training, testing and validation datasets. We strongly belive that hold-out validation is the most redundant validation method and although miRNAselector supports cross-validation, the hold-out validation is set by default in most cases. Thus, the rest of the analysis is dependent of existance of 3 seperate datasets:
mixed_train.csv): By default 60%, used for differential expression, used for feature selection, used for model training.mixed_test.csv): By default 20%, used for hyperparameter selection (in holdout=T mode), used for performance assessment.mixed_valid.csv): By default 20%, used only for performance assessment.The best signiture (best set of miRNAs for diagnostic test) can be selected based on all 3 datasets, 2 datasets or only validation set. The process of best signiture selection will be discussed below.
The split can be prepared manually by user (the pipeline expects to find mixed_*.csv files in working directory) or in a convinient way using ks.prepare_split(). Let’s do it now.
| test | train | valid | |
|---|---|---|---|
| Cancer | 36 | 107 | 35 |
| Control | 36 | 107 | 35 |
| hsa.miR.30c.5p | hsa.miR.30c.2.3p | hsa.miR.30d.5p | hsa.miR.139.5p | hsa.miR.10a.5p | hsa.miR.10b.5p | Class.Class |
|---|---|---|---|---|---|---|
| 2.598949 | 1.514444 | 3.802908 | 1.555401 | 4.735516 | 4.298381 | Cancer |
| 2.391663 | 1.416540 | 3.656058 | 1.326116 | 4.615369 | 3.946954 | Cancer |
| 2.576255 | 1.566166 | 3.721311 | 1.220338 | 4.385001 | 3.998914 | Cancer |
| 2.260492 | 1.401010 | 3.539243 | 1.924365 | 3.945605 | 4.486228 | Cancer |
| 2.595315 | 1.562735 | 3.571441 | 1.721510 | 4.519249 | 4.126032 | Cancer |
| 2.632860 | 1.723000 | 3.708020 | 2.120359 | 4.875796 | 3.887056 | Cancer |
| 2.747742 | 2.016887 | 3.852090 | 1.533525 | 3.961177 | 3.642831 | Cancer |
| 3.037578 | 1.880192 | 3.960987 | 1.798007 | 4.675070 | 4.133894 | Cancer |
| 2.598933 | 1.291485 | 3.581714 | 1.413095 | 4.580566 | 3.960672 | Cancer |
| 2.829601 | 1.494520 | 3.589292 | 1.755581 | 4.687104 | 4.209582 | Cancer |
We can see that the dataset was devided in balanced way. Now we are ready to move to the analysis…
In biomarker studies we relay on validation. We perform hold-out validation, but the signature has to be selected based on trainin dataset only. Including testing and validation dataset in the exploratory analysis could lead to bias. In the following section we show how to use our package to perform quick exploratory analysis of miRNA-seq data.
dane = ks.load_datamix(use_smote_not_rose = T) # load mixed_*.csv files
train = dane[[1]]
test = dane[[2]]
valid = dane[[3]]
train_smoted = dane[[4]]
trainx = dane[[5]]
trainx_smoted = dane[[6]] # get the objects from list to make the code more readable.ks_load_datamix() function loads the data created in preparation phase. It requires the output constructed by ks.prepare_split() function to be placed in working directory (‘wd’), thus files ‘mixed_train.csv’, ‘mixed_test.csv’ and ‘mixed_valid.csv’ have to exist in the directory. For imbalanced data, the fuction can perform balancing using:
At the beging of the analysis we usually perform principal component analysis (PCA) to assess for any batch effect, possible outliers and get a general understanding of miRNA profile. This package can construct 2-dimentional biplot and 3-dimentional scatterplot based on the computed components to handle this issue.
In the next step we can correct the batch effect for example using ks.combat(). The correction of batch effect is out of the scope of this tutorial.
Usually, the next step in the exploratory analysis is to perform the differential expression analysis. Differential expression in our package focuces of biomarker discovery thus uses t-test with the correction for multiple comparisons. The following table shows signifiant miRNAs after BH correction.
de = ks.miRNA_differential_expression(trainx, train$Class)
sig_de = de %>% dplyr::filter(`p-value BH` <= 0.05) %>% dplyr::arrange(`p-value BH`) # leave only significant after Benjamini-Hochberg procedure and sort by ascending p-value
ks.table(sig_de)| miR | mean logtpm | median logtpm | SD logtpm | cancer mean | cancer median | cancer SD | control mean | control median | control SD | log10FC (subtr estim) | log10FC | log2FC | reversed_log10FC | reverse_log2FC | p-value | p-value Bonferroni | p-value Holm | -log10(p-value Bonferroni) | p-value BH |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| hsa.miR.21.5p | 5.081371 | 5.120977 | 0.5409456 | 5.473043 | 5.559281 | 0.3042087 | 4.689698 | 4.744446 | 0.4309906 | -0.7833450 | 0.7833450 | 2.6022157 | 0.7833450 | 2.6022157 | 0.0000000 | 0.0000000 | 0.0000000 | 32.2312221 | 0.0000000 |
| hsa.miR.30e.3p | 3.654915 | 3.620465 | 0.2922656 | 3.445100 | 3.469065 | 0.1835853 | 3.864731 | 3.891680 | 0.2215066 | 0.4196304 | -0.4196304 | -1.3939819 | -0.4196304 | -1.3939819 | 0.0000000 | 0.0000000 | 0.0000000 | 32.1422635 | 0.0000000 |
| hsa.miR.30a.3p | 3.861758 | 3.760500 | 0.5107622 | 3.494206 | 3.528067 | 0.2957390 | 4.229311 | 4.315264 | 0.4049904 | 0.7351047 | -0.7351047 | -2.4419651 | -0.7351047 | -2.4419651 | 0.0000000 | 0.0000000 | 0.0000000 | 31.8224321 | 0.0000000 |
| hsa.miR.30c.2.3p | 1.791478 | 1.685323 | 0.4609305 | 1.478041 | 1.473434 | 0.2548946 | 2.104915 | 2.062559 | 0.4044845 | 0.6268741 | -0.6268741 | -2.0824306 | -0.6268741 | -2.0824306 | 0.0000000 | 0.0000000 | 0.0000000 | 26.3237118 | 0.0000000 |
| hsa.miR.132.3p | 2.062941 | 2.061854 | 0.3206914 | 2.268277 | 2.222890 | 0.2453178 | 1.857606 | 1.842071 | 0.2476985 | -0.4106709 | 0.4106709 | 1.3642193 | 0.4106709 | 1.3642193 | 0.0000000 | 0.0000000 | 0.0000000 | 23.2892084 | 0.0000000 |
| hsa.miR.375.3p | 3.851381 | 4.088129 | 1.1312626 | 4.572062 | 4.593208 | 0.6167635 | 3.130699 | 3.420583 | 1.0689350 | -1.4413633 | 1.4413633 | 4.7881053 | 1.4413633 | 4.7881053 | 0.0000000 | 0.0000000 | 0.0000000 | 21.6930447 | 0.0000000 |
| hsa.miR.217.5p | 2.065790 | 1.932856 | 1.2184284 | 2.817892 | 2.596188 | 1.0079357 | 1.313689 | 1.247738 | 0.9084168 | -1.5042026 | 1.5042026 | 4.9968529 | 1.5042026 | 4.9968529 | 0.0000000 | 0.0000000 | 0.0000000 | 21.0021448 | 0.0000000 |
| hsa.miR.139.5p | 1.916157 | 1.919751 | 0.4638487 | 1.634747 | 1.558070 | 0.3573670 | 2.197567 | 2.204857 | 0.3804627 | 0.5628209 | -0.5628209 | -1.8696506 | -0.5628209 | -1.8696506 | 0.0000000 | 0.0000000 | 0.0000000 | 20.0671420 | 0.0000000 |
| hsa.miR.199a.5p | 3.034892 | 3.117770 | 0.3929526 | 3.267832 | 3.318017 | 0.2924584 | 2.801951 | 2.808411 | 0.3394148 | -0.4658802 | 0.4658802 | 1.5476204 | 0.4658802 | 1.5476204 | 0.0000000 | 0.0000000 | 0.0000000 | 18.7831163 | 0.0000000 |
| hsa.miR.338.3p | 2.743796 | 2.778247 | 0.4190933 | 2.991348 | 2.994550 | 0.3219668 | 2.496243 | 2.539407 | 0.3543332 | -0.4951049 | 0.4951049 | 1.6447027 | 0.4951049 | 1.6447027 | 0.0000000 | 0.0000000 | 0.0000000 | 18.6530266 | 0.0000000 |
| hsa.miR.199b.5p | 2.146095 | 2.358829 | 0.6145137 | 2.504540 | 2.573683 | 0.3567535 | 1.787650 | 1.799962 | 0.6100440 | -0.7168901 | 0.7168901 | 2.3814574 | 0.7168901 | 2.3814574 | 0.0000000 | 0.0000000 | 0.0000000 | 17.2507100 | 0.0000000 |
| hsa.miR.127.5p | 2.214220 | 2.327709 | 0.4803146 | 2.486170 | 2.478848 | 0.3020641 | 1.942270 | 1.939357 | 0.4722596 | -0.5438996 | 0.5438996 | 1.8067954 | 0.5438996 | 1.8067954 | 0.0000000 | 0.0000000 | 0.0000000 | 16.1634770 | 0.0000000 |
| hsa.miR.381.3p | 1.720930 | 1.791253 | 0.4579339 | 1.974984 | 1.949285 | 0.3353016 | 1.466877 | 1.476716 | 0.4226746 | -0.5081072 | 0.5081072 | 1.6878957 | 0.5081072 | 1.6878957 | 0.0000000 | 0.0000000 | 0.0000000 | 15.6751972 | 0.0000000 |
| hsa.miR.127.3p | 2.789656 | 2.829601 | 0.3874414 | 2.995898 | 3.006282 | 0.3147991 | 2.583413 | 2.568116 | 0.3415622 | -0.4124850 | 0.4124850 | 1.3702456 | 0.4124850 | 1.3702456 | 0.0000000 | 0.0000000 | 0.0000000 | 14.1819049 | 0.0000000 |
| hsa.miR.199a.3p | 3.431280 | 3.533249 | 0.3595668 | 3.620030 | 3.670480 | 0.2685999 | 3.242530 | 3.285480 | 0.3401837 | -0.3775004 | 0.3775004 | 1.2540293 | 0.3775004 | 1.2540293 | 0.0000000 | 0.0000000 | 0.0000000 | 13.5707699 | 0.0000000 |
| hsa.miR.181b.5p | 2.447025 | 2.523360 | 0.4131408 | 2.664941 | 2.705753 | 0.2848923 | 2.229109 | 2.205351 | 0.4073655 | -0.4358316 | 0.4358316 | 1.4478013 | 0.4358316 | 1.4478013 | 0.0000000 | 0.0000000 | 0.0000000 | 13.5990480 | 0.0000000 |
| hsa.miR.199b.3p | 3.430095 | 3.531945 | 0.3598114 | 3.619075 | 3.669630 | 0.2686093 | 3.241115 | 3.284615 | 0.3404382 | -0.3779599 | 0.3779599 | 1.2555557 | 0.3779599 | 1.2555557 | 0.0000000 | 0.0000000 | 0.0000000 | 13.5889555 | 0.0000000 |
| hsa.miR.194.5p | 3.119571 | 3.359054 | 0.8329947 | 3.562675 | 3.633752 | 0.4167141 | 2.676468 | 2.410622 | 0.9078933 | -0.8862067 | 0.8862067 | 2.9439150 | 0.8862067 | 2.9439150 | 0.0000000 | 0.0000000 | 0.0000000 | 13.2390851 | 0.0000000 |
| hsa.miR.134.5p | 2.366486 | 2.433393 | 0.4088244 | 2.572458 | 2.557931 | 0.2993653 | 2.160514 | 2.115254 | 0.4007277 | -0.4119434 | 0.4119434 | 1.3684465 | 0.4119434 | 1.3684465 | 0.0000000 | 0.0000000 | 0.0000000 | 12.1552043 | 0.0000000 |
| hsa.miR.30a.5p | 4.429837 | 4.356060 | 0.4519742 | 4.199892 | 4.226049 | 0.2683567 | 4.659782 | 4.685628 | 0.4813796 | 0.4598896 | -0.4598896 | -1.5277200 | -0.4598896 | -1.5277200 | 0.0000000 | 0.0000000 | 0.0000000 | 12.1052493 | 0.0000000 |
| hsa.miR.200a.5p | 2.480123 | 2.527551 | 0.5262405 | 2.744628 | 2.777438 | 0.3622178 | 2.215619 | 2.330898 | 0.5329388 | -0.5290086 | 0.5290086 | 1.7573286 | 0.5290086 | 1.7573286 | 0.0000000 | 0.0000000 | 0.0000000 | 11.9837197 | 0.0000000 |
| hsa.miR.136.5p | 1.426048 | 1.539528 | 0.5852494 | 1.722033 | 1.713813 | 0.2950712 | 1.130063 | 1.259715 | 0.6514071 | -0.5919692 | 0.5919692 | 1.9664791 | 0.5919692 | 1.9664791 | 0.0000000 | 0.0000000 | 0.0000000 | 11.6576372 | 0.0000000 |
| hsa.miR.192.5p | 3.474000 | 3.723466 | 0.8856187 | 3.920081 | 3.942126 | 0.4108526 | 3.027920 | 2.831424 | 1.0027511 | -0.8921615 | 0.8921615 | 2.9636965 | 0.8921615 | 2.9636965 | 0.0000000 | 0.0000000 | 0.0000000 | 11.4242384 | 0.0000000 |
| hsa.miR.451a | 2.540774 | 2.525599 | 0.6339915 | 2.233131 | 2.192706 | 0.4710976 | 2.848417 | 2.906464 | 0.6282318 | 0.6152860 | -0.6152860 | -2.0439357 | -0.6152860 | -2.0439357 | 0.0000000 | 0.0000000 | 0.0000000 | 11.0384525 | 0.0000000 |
| hsa.miR.200a.3p | 2.330223 | 2.419049 | 0.6863313 | 2.660594 | 2.674894 | 0.3970887 | 1.999851 | 2.171325 | 0.7539997 | -0.6607432 | 0.6607432 | 2.1949415 | 0.6607432 | 2.1949415 | 0.0000000 | 0.0000000 | 0.0000000 | 10.4637303 | 0.0000000 |
| hsa.miR.141.3p | 2.470222 | 2.587047 | 0.6758787 | 2.787653 | 2.835787 | 0.4142754 | 2.152790 | 2.366831 | 0.7368061 | -0.6348621 | 0.6348621 | 2.1089664 | 0.6348621 | 2.1089664 | 0.0000000 | 0.0000000 | 0.0000000 | 9.8966505 | 0.0000000 |
| hsa.miR.181c.3p | 1.549848 | 1.598609 | 0.3058360 | 1.691125 | 1.690791 | 0.2009568 | 1.408572 | 1.397065 | 0.3275293 | -0.2825529 | 0.2825529 | 0.9386203 | 0.2825529 | 0.9386203 | 0.0000000 | 0.0000000 | 0.0000000 | 9.5661290 | 0.0000000 |
| hsa.miR.30d.5p | 3.824938 | 3.786939 | 0.2585958 | 3.705810 | 3.708020 | 0.1763818 | 3.944065 | 3.950384 | 0.2731545 | 0.2382549 | -0.2382549 | -0.7914657 | -0.2382549 | -0.7914657 | 0.0000000 | 0.0000000 | 0.0000000 | 9.5425303 | 0.0000000 |
| hsa.miR.200b.3p | 2.716967 | 2.769036 | 0.5121158 | 2.949316 | 3.014389 | 0.3627291 | 2.484618 | 2.625241 | 0.5351976 | -0.4646977 | 0.4646977 | 1.5436922 | 0.4646977 | 1.5436922 | 0.0000000 | 0.0000000 | 0.0000000 | 9.2128238 | 0.0000000 |
| hsa.miR.708.3p | 1.553047 | 1.799718 | 0.7046857 | 1.878493 | 1.937453 | 0.3169286 | 1.227601 | 1.186108 | 0.8267867 | -0.6508920 | 0.6508920 | 2.1622164 | 0.6508920 | 2.1622164 | 0.0000000 | 0.0000000 | 0.0000000 | 9.1585590 | 0.0000000 |
| hsa.miR.210.3p | 2.239766 | 2.271507 | 0.6889295 | 2.547543 | 2.542301 | 0.5385826 | 1.931988 | 2.106523 | 0.6873228 | -0.6155552 | 0.6155552 | 2.0448301 | 0.6155552 | 2.0448301 | 0.0000000 | 0.0000000 | 0.0000000 | 8.9396288 | 0.0000000 |
| hsa.miR.486.5p | 2.102968 | 2.062283 | 0.6191482 | 1.830616 | 1.754483 | 0.4121882 | 2.375321 | 2.342782 | 0.6713086 | 0.5447046 | -0.5447046 | -1.8094696 | -0.5447046 | -1.8094696 | 0.0000000 | 0.0000000 | 0.0000000 | 8.4375052 | 0.0000000 |
| hsa.miR.29a.3p | 3.841416 | 3.818064 | 0.2415932 | 3.736185 | 3.745283 | 0.1796643 | 3.946646 | 3.954986 | 0.2503002 | 0.2104605 | -0.2104605 | -0.6991346 | -0.2104605 | -0.6991346 | 0.0000000 | 0.0000000 | 0.0000000 | 8.3240291 | 0.0000000 |
| hsa.miR.378a.3p | 2.550650 | 2.512377 | 0.3937521 | 2.378397 | 2.381114 | 0.2557418 | 2.722903 | 2.778885 | 0.4315532 | 0.3445063 | -0.3445063 | -1.1444252 | -0.3445063 | -1.1444252 | 0.0000000 | 0.0000000 | 0.0000000 | 8.2934009 | 0.0000000 |
| hsa.miR.379.5p | 2.828974 | 2.918613 | 0.4091041 | 3.006836 | 3.034753 | 0.2966347 | 2.651112 | 2.644205 | 0.4294792 | -0.3557240 | 0.3557240 | 1.1816895 | 0.3557240 | 1.1816895 | 0.0000000 | 0.0000000 | 0.0000000 | 8.2616630 | 0.0000000 |
| hsa.miR.141.5p | 2.216028 | 2.392275 | 0.6525748 | 2.500775 | 2.556782 | 0.3976527 | 1.931282 | 2.148570 | 0.7306892 | -0.5694928 | 0.5694928 | 1.8918140 | 0.5694928 | 1.8918140 | 0.0000000 | 0.0000000 | 0.0000000 | 8.1778863 | 0.0000000 |
| hsa.miR.21.3p | 2.950511 | 3.002843 | 0.4495447 | 3.143884 | 3.238660 | 0.3956108 | 2.757137 | 2.698429 | 0.4172376 | -0.3867473 | 0.3867473 | 1.2847466 | 0.3867473 | 1.2847466 | 0.0000000 | 0.0000000 | 0.0000000 | 8.1503082 | 0.0000000 |
| hsa.miR.337.3p | 1.498952 | 1.609902 | 0.4286748 | 1.685506 | 1.704399 | 0.2599192 | 1.312398 | 1.278140 | 0.4810808 | -0.3731072 | 0.3731072 | 1.2394354 | 0.3731072 | 1.2394354 | 0.0000000 | 0.0000000 | 0.0000000 | 8.1171714 | 0.0000000 |
| hsa.miR.144.5p | 1.799486 | 1.840358 | 0.7365132 | 1.481479 | 1.461856 | 0.4729441 | 2.117492 | 2.202314 | 0.8137450 | 0.6360122 | -0.6360122 | -2.1127868 | -0.6360122 | -2.1127868 | 0.0000000 | 0.0000000 | 0.0000000 | 8.0043727 | 0.0000000 |
| hsa.miR.429 | 1.847904 | 1.950131 | 0.7268633 | 2.162646 | 2.205144 | 0.3857231 | 1.533162 | 1.702486 | 0.8443154 | -0.6294838 | 0.6294838 | 2.0910999 | 0.6294838 | 2.0910999 | 0.0000000 | 0.0000000 | 0.0000000 | 7.8988604 | 0.0000000 |
| hsa.miR.335.3p | 1.773607 | 1.812886 | 0.3718545 | 1.932716 | 1.892784 | 0.2426583 | 1.614497 | 1.631930 | 0.4097115 | -0.3182195 | 0.3182195 | 1.0571023 | 0.3182195 | 1.0571023 | 0.0000000 | 0.0000000 | 0.0000000 | 7.8314098 | 0.0000000 |
| hsa.miR.126.5p | 2.232491 | 2.222927 | 0.4152358 | 2.056938 | 2.063790 | 0.2104761 | 2.408045 | 2.529680 | 0.4898455 | 0.3511067 | -0.3511067 | -1.1663511 | -0.3511067 | -1.1663511 | 0.0000000 | 0.0000000 | 0.0000000 | 7.3936205 | 0.0000000 |
| hsa.miR.30c.5p | 2.808698 | 2.749100 | 0.3490929 | 2.661967 | 2.661797 | 0.1963921 | 2.955429 | 2.915562 | 0.4035411 | 0.2934616 | -0.2934616 | -0.9748585 | -0.2934616 | -0.9748585 | 0.0000000 | 0.0000000 | 0.0000000 | 7.3565035 | 0.0000000 |
| hsa.miR.181a.3p | 2.106380 | 2.174268 | 0.3784895 | 2.261578 | 2.305816 | 0.2347854 | 1.951183 | 2.028792 | 0.4290792 | -0.3103947 | 0.3103947 | 1.0311087 | 0.3103947 | 1.0311087 | 0.0000000 | 0.0000001 | 0.0000001 | 6.9641475 | 0.0000000 |
| hsa.let.7c.5p | 3.419322 | 3.422382 | 0.3296842 | 3.287237 | 3.332260 | 0.2860501 | 3.551407 | 3.535984 | 0.3183740 | 0.2641699 | -0.2641699 | -0.8775535 | -0.2641699 | -0.8775535 | 0.0000000 | 0.0000002 | 0.0000001 | 6.7432319 | 0.0000000 |
| hsa.miR.181a.5p | 3.155295 | 3.246986 | 0.4050583 | 3.316282 | 3.343112 | 0.2324404 | 2.994307 | 2.991241 | 0.4725895 | -0.3219750 | 0.3219750 | 1.0695778 | 0.3219750 | 1.0695778 | 0.0000000 | 0.0000004 | 0.0000003 | 6.3623836 | 0.0000000 |
| hsa.miR.221.3p | 2.290528 | 2.336670 | 0.3527519 | 2.427381 | 2.408816 | 0.2861108 | 2.153674 | 2.182603 | 0.3610694 | -0.2737072 | 0.2737072 | 0.9092357 | 0.2737072 | 0.9092357 | 0.0000000 | 0.0000007 | 0.0000005 | 6.1577028 | 0.0000000 |
| hsa.miR.93.5p | 3.298589 | 3.328666 | 0.2589551 | 3.398683 | 3.400925 | 0.1891085 | 3.198495 | 3.236571 | 0.2806412 | -0.2001879 | 0.2001879 | 0.6650098 | 0.2001879 | 0.6650098 | 0.0000000 | 0.0000009 | 0.0000006 | 6.0433338 | 0.0000000 |
| hsa.miR.378a.5p | 1.765728 | 1.699123 | 0.3998851 | 1.610886 | 1.612851 | 0.2430057 | 1.920570 | 1.955764 | 0.4624594 | 0.3096836 | -0.3096836 | -1.0287466 | -0.3096836 | -1.0287466 | 0.0000000 | 0.0000011 | 0.0000008 | 5.9662895 | 0.0000000 |
| hsa.miR.146b.5p | 2.601785 | 2.674141 | 0.4626849 | 2.779955 | 2.835954 | 0.2919913 | 2.423615 | 2.450139 | 0.5299317 | -0.3563393 | 0.3563393 | 1.1837335 | 0.3563393 | 1.1837335 | 0.0000000 | 0.0000013 | 0.0000009 | 5.8990220 | 0.0000000 |
| hsa.miR.23a.3p | 3.411765 | 3.464466 | 0.2170216 | 3.494149 | 3.504609 | 0.1702973 | 3.329380 | 3.342163 | 0.2278979 | -0.1647689 | 0.1647689 | 0.5473504 | 0.1647689 | 0.5473504 | 0.0000000 | 0.0000016 | 0.0000011 | 5.7880964 | 0.0000000 |
| hsa.miR.30b.5p | 2.622413 | 2.622930 | 0.3212403 | 2.499914 | 2.507222 | 0.2120817 | 2.744913 | 2.792110 | 0.3634422 | 0.2449988 | -0.2449988 | -0.8138683 | -0.2449988 | -0.8138683 | 0.0000000 | 0.0000017 | 0.0000012 | 5.7695073 | 0.0000000 |
| hsa.miR.625.3p | 2.068354 | 2.104397 | 0.3502308 | 2.200772 | 2.210917 | 0.2188272 | 1.935936 | 1.910435 | 0.4039743 | -0.2648357 | 0.2648357 | 0.8797652 | 0.2648357 | 0.8797652 | 0.0000000 | 0.0000025 | 0.0000017 | 5.6075234 | 0.0000000 |
| hsa.miR.182.5p | 3.674023 | 3.755735 | 0.4922991 | 3.858145 | 3.842168 | 0.3744427 | 3.489902 | 3.616382 | 0.5275933 | -0.3682429 | 0.3682429 | 1.2232765 | 0.3682429 | 1.2232765 | 0.0000000 | 0.0000029 | 0.0000020 | 5.5415728 | 0.0000001 |
| hsa.miR.27a.3p | 3.005903 | 3.044024 | 0.2647920 | 3.104509 | 3.109418 | 0.2287859 | 2.907296 | 2.903005 | 0.2625226 | -0.1972135 | 0.1972135 | 0.6551290 | 0.1972135 | 0.6551290 | 0.0000000 | 0.0000030 | 0.0000020 | 5.5228315 | 0.0000001 |
| hsa.miR.183.5p | 3.203446 | 3.257163 | 0.4833569 | 3.381328 | 3.357706 | 0.3966813 | 3.025563 | 3.143805 | 0.4982314 | -0.3557648 | 0.3557648 | 1.1818251 | 0.3557648 | 1.1818251 | 0.0000000 | 0.0000047 | 0.0000031 | 5.3289624 | 0.0000001 |
| hsa.miR.181c.5p | 1.665600 | 1.682222 | 0.3601516 | 1.795267 | 1.778815 | 0.2462991 | 1.535932 | 1.501756 | 0.4074728 | -0.2593354 | 0.2593354 | 0.8614935 | 0.2593354 | 0.8614935 | 0.0000001 | 0.0000115 | 0.0000076 | 4.9383329 | 0.0000002 |
| hsa.miR.126.3p | 3.305734 | 3.272928 | 0.4223713 | 3.157448 | 3.162873 | 0.2190488 | 3.454020 | 3.630912 | 0.5158520 | 0.2965719 | -0.2965719 | -0.9851906 | -0.2965719 | -0.9851906 | 0.0000002 | 0.0000319 | 0.0000209 | 4.4963337 | 0.0000005 |
| hsa.miR.582.3p | 2.087755 | 2.119436 | 0.3376004 | 1.973266 | 1.974597 | 0.2964765 | 2.202245 | 2.291615 | 0.3386179 | 0.2289783 | -0.2289783 | -0.7606496 | -0.2289783 | -0.7606496 | 0.0000004 | 0.0000584 | 0.0000380 | 4.2335770 | 0.0000010 |
| hsa.miR.143.3p | 4.940749 | 4.860769 | 0.4133211 | 4.799815 | 4.796805 | 0.2682101 | 5.081683 | 4.987285 | 0.4808783 | 0.2818686 | -0.2818686 | -0.9363472 | -0.2818686 | -0.9363472 | 0.0000004 | 0.0000618 | 0.0000398 | 4.2090891 | 0.0000010 |
| hsa.miR.185.5p | 1.698820 | 1.752167 | 0.3122312 | 1.803017 | 1.789962 | 0.1825995 | 1.594623 | 1.585711 | 0.3750133 | -0.2083937 | 0.2083937 | 0.6922687 | 0.2083937 | 0.6922687 | 0.0000007 | 0.0001204 | 0.0000769 | 3.9192525 | 0.0000020 |
| hsa.miR.29c.3p | 3.321214 | 3.327217 | 0.3591607 | 3.202955 | 3.215627 | 0.2277490 | 3.439473 | 3.491354 | 0.4232090 | 0.2365180 | -0.2365180 | -0.7856959 | -0.2365180 | -0.7856959 | 0.0000010 | 0.0001621 | 0.0001026 | 3.7900871 | 0.0000026 |
| hsa.miR.223.3p | 2.257390 | 2.288581 | 0.4632869 | 2.408097 | 2.462940 | 0.4003810 | 2.106684 | 2.111760 | 0.4744849 | -0.3014129 | 0.3014129 | 1.0012719 | 0.3014129 | 1.0012719 | 0.0000011 | 0.0001834 | 0.0001149 | 3.7365180 | 0.0000029 |
| hsa.miR.423.5p | 1.545436 | 1.538957 | 0.3144486 | 1.443133 | 1.425446 | 0.2069216 | 1.647738 | 1.666376 | 0.3670736 | 0.2046048 | -0.2046048 | -0.6796823 | -0.2046048 | -0.6796823 | 0.0000013 | 0.0002154 | 0.0001337 | 3.6667565 | 0.0000034 |
| hsa.miR.151a.5p | 1.778564 | 1.824896 | 0.2355297 | 1.854691 | 1.836098 | 0.1493639 | 1.702436 | 1.739749 | 0.2783202 | -0.1522549 | 0.1522549 | 0.5057800 | 0.1522549 | 0.5057800 | 0.0000016 | 0.0002604 | 0.0001600 | 3.5843739 | 0.0000040 |
| hsa.let.7i.5p | 2.599921 | 2.632392 | 0.2997045 | 2.696086 | 2.717300 | 0.1738861 | 2.503755 | 2.466912 | 0.3627486 | -0.1923305 | 0.1923305 | 0.6389080 | 0.1923305 | 0.6389080 | 0.0000020 | 0.0003296 | 0.0002005 | 3.4820413 | 0.0000050 |
| hsa.miR.26b.5p | 2.777405 | 2.755033 | 0.2942712 | 2.683523 | 2.667076 | 0.2076405 | 2.871288 | 2.871957 | 0.3363026 | 0.1877642 | -0.1877642 | -0.6237392 | -0.1877642 | -0.6237392 | 0.0000020 | 0.0003363 | 0.0002026 | 3.4732893 | 0.0000050 |
| hsa.miR.222.3p | 1.679643 | 1.690400 | 0.3421027 | 1.788188 | 1.797616 | 0.2933599 | 1.571098 | 1.634996 | 0.3540146 | -0.2170897 | 0.2170897 | 0.7211563 | 0.2170897 | 0.7211563 | 0.0000021 | 0.0003468 | 0.0002068 | 3.4599760 | 0.0000051 |
| hsa.miR.320a.3p | 2.517259 | 2.510587 | 0.2335402 | 2.443505 | 2.440882 | 0.2043264 | 2.591012 | 2.575436 | 0.2384649 | 0.1475073 | -0.1475073 | -0.4900086 | -0.1475073 | -0.4900086 | 0.0000023 | 0.0003863 | 0.0002281 | 3.4130330 | 0.0000056 |
| hsa.miR.30e.5p | 3.679150 | 3.640665 | 0.3123538 | 3.579560 | 3.581958 | 0.1513627 | 3.778740 | 3.843580 | 0.3913006 | 0.1991799 | -0.1991799 | -0.6616613 | -0.1991799 | -0.6616613 | 0.0000025 | 0.0004221 | 0.0002467 | 3.3745491 | 0.0000060 |
| hsa.miR.200c.3p | 3.479500 | 3.647500 | 0.6328315 | 3.679499 | 3.713871 | 0.3714689 | 3.279502 | 3.524250 | 0.7654977 | -0.3999973 | 0.3999973 | 1.3287624 | 0.3999973 | 1.3287624 | 0.0000028 | 0.0004717 | 0.0002728 | 3.3263079 | 0.0000066 |
| hsa.miR.145.5p | 3.405874 | 3.362197 | 0.3557557 | 3.294171 | 3.279622 | 0.2492612 | 3.517576 | 3.469948 | 0.4086525 | 0.2234046 | -0.2234046 | -0.7421341 | -0.2234046 | -0.7421341 | 0.0000030 | 0.0004969 | 0.0002844 | 3.3037592 | 0.0000069 |
| hsa.miR.92b.3p | 1.748375 | 1.731752 | 0.3770581 | 1.865526 | 1.808052 | 0.2617419 | 1.631225 | 1.619889 | 0.4352821 | -0.2343014 | 0.2343014 | 0.7783325 | 0.2343014 | 0.7783325 | 0.0000039 | 0.0006400 | 0.0003624 | 3.1938498 | 0.0000088 |
| hsa.miR.629.5p | 1.718671 | 1.767577 | 0.2822432 | 1.804344 | 1.819456 | 0.2016194 | 1.632997 | 1.650836 | 0.3234272 | -0.1713470 | 0.1713470 | 0.5692024 | 0.1713470 | 0.5692024 | 0.0000064 | 0.0010697 | 0.0005993 | 2.9707322 | 0.0000145 |
| hsa.miR.99a.5p | 2.916024 | 2.912899 | 0.3862114 | 2.800660 | 2.801168 | 0.3181101 | 3.031388 | 3.020665 | 0.4143220 | 0.2307282 | -0.2307282 | -0.7664624 | -0.2307282 | -0.7664624 | 0.0000086 | 0.0014262 | 0.0007904 | 2.8458122 | 0.0000190 |
| hsa.miR.22.3p | 4.901288 | 4.935034 | 0.2599745 | 4.979132 | 4.983023 | 0.1344526 | 4.823443 | 4.825832 | 0.3248074 | -0.1556883 | 0.1556883 | 0.5171853 | 0.1556883 | 0.5171853 | 0.0000101 | 0.0016706 | 0.0009158 | 2.7771206 | 0.0000220 |
| hsa.miR.10b.5p | 4.284825 | 4.160990 | 0.6577961 | 4.092498 | 4.126032 | 0.2208435 | 4.477153 | 4.339191 | 0.8637267 | 0.3846552 | -0.3846552 | -1.2777969 | -0.3846552 | -1.2777969 | 0.0000184 | 0.0030501 | 0.0016537 | 2.5156825 | 0.0000396 |
| hsa.miR.142.3p | 2.908082 | 2.993345 | 0.5523317 | 3.062898 | 3.069081 | 0.3324835 | 2.753266 | 2.754739 | 0.6738583 | -0.3096321 | 0.3096321 | 1.0285755 | 0.3096321 | 1.0285755 | 0.0000350 | 0.0058148 | 0.0031176 | 2.2354673 | 0.0000745 |
| hsa.miR.16.5p | 2.418426 | 2.396977 | 0.2678058 | 2.344887 | 2.354009 | 0.1520643 | 2.491965 | 2.532161 | 0.3317754 | 0.1470782 | -0.1470782 | -0.4885833 | -0.1470782 | -0.4885833 | 0.0000518 | 0.0086050 | 0.0045617 | 2.0652508 | 0.0001089 |
| hsa.miR.10a.5p | 4.453193 | 4.572159 | 0.4172756 | 4.567356 | 4.619375 | 0.2726925 | 4.339031 | 4.448143 | 0.4992066 | -0.2283251 | 0.2283251 | 0.7584795 | 0.2283251 | 0.7584795 | 0.0000528 | 0.0087721 | 0.0045974 | 2.0568959 | 0.0001097 |
| hsa.miR.365a.3p | 1.558982 | 1.524607 | 0.3288053 | 1.472455 | 1.482687 | 0.2603646 | 1.645509 | 1.646930 | 0.3665257 | 0.1730535 | -0.1730535 | -0.5748714 | -0.1730535 | -0.5748714 | 0.0000972 | 0.0161318 | 0.0083574 | 1.7923179 | 0.0001970 |
| hsa.miR.365b.3p | 1.559072 | 1.524607 | 0.3286232 | 1.472600 | 1.482687 | 0.2598729 | 1.645545 | 1.646930 | 0.3665720 | 0.1729456 | -0.1729456 | -0.5745130 | -0.1729456 | -0.5745130 | 0.0000973 | 0.0161562 | 0.0083574 | 1.7916615 | 0.0001970 |
| hsa.miR.20a.5p | 2.167170 | 2.188072 | 0.3531756 | 2.258504 | 2.252491 | 0.2151305 | 2.075836 | 2.092815 | 0.4330362 | -0.1826683 | 0.1826683 | 0.6068109 | 0.1826683 | 0.6068109 | 0.0001387 | 0.0230266 | 0.0116520 | 1.6377698 | 0.0002774 |
| hsa.miR.195.5p | 1.610326 | 1.644712 | 0.4593292 | 1.492892 | 1.530241 | 0.2351047 | 1.727760 | 1.829698 | 0.5838159 | 0.2348685 | -0.2348685 | -0.7802161 | -0.2348685 | -0.7802161 | 0.0001727 | 0.0286635 | 0.0143318 | 1.5426701 | 0.0003412 |
| hsa.miR.26a.5p | 3.363368 | 3.336961 | 0.2249507 | 3.307628 | 3.297116 | 0.1471895 | 3.419108 | 3.407502 | 0.2715615 | 0.1114806 | -0.1114806 | -0.3703306 | -0.1114806 | -0.3703306 | 0.0002606 | 0.0432635 | 0.0213711 | 1.3638783 | 0.0005090 |
| hsa.miR.145.3p | 1.779374 | 1.754841 | 0.2975307 | 1.706504 | 1.693139 | 0.2239980 | 1.852244 | 1.840403 | 0.3420364 | 0.1457395 | -0.1457395 | -0.4841363 | -0.1457395 | -0.4841363 | 0.0002988 | 0.0495960 | 0.0242005 | 1.3045533 | 0.0005767 |
| hsa.miR.142.5p | 1.517108 | 1.642632 | 0.7709832 | 1.707036 | 1.728336 | 0.3336782 | 1.327180 | 1.413437 | 1.0051222 | -0.3798562 | 0.3798562 | 1.2618548 | 0.3798562 | 1.2618548 | 0.0003066 | 0.0509030 | 0.0245316 | 1.2932569 | 0.0005851 |
| hsa.miR.155.5p | 2.133480 | 2.195366 | 0.4195203 | 2.233687 | 2.297068 | 0.3475670 | 2.033272 | 2.006969 | 0.4610634 | -0.2004154 | 0.2004154 | 0.6657657 | 0.2004154 | 0.6657657 | 0.0004167 | 0.0691679 | 0.0329172 | 1.1600957 | 0.0007860 |
| hsa.let.7d.5p | 2.204108 | 2.185887 | 0.2122742 | 2.154807 | 2.147458 | 0.1420691 | 2.253409 | 2.269843 | 0.2558420 | 0.0986016 | -0.0986016 | -0.3275473 | -0.0986016 | -0.3275473 | 0.0006292 | 0.1044446 | 0.0490764 | 0.9811142 | 0.0011735 |
| hsa.miR.101.3p | 4.064812 | 4.041831 | 0.3932389 | 3.975163 | 3.934040 | 0.1680979 | 4.154461 | 4.258030 | 0.5159943 | 0.1792983 | -0.1792983 | -0.5956161 | -0.1792983 | -0.5956161 | 0.0008471 | 0.1406149 | 0.0652250 | 0.8519688 | 0.0015624 |
| hsa.miR.15a.5p | 1.962872 | 1.985385 | 0.2869339 | 2.027709 | 2.007988 | 0.1768348 | 1.898034 | 1.940992 | 0.3545154 | -0.1296752 | 0.1296752 | 0.4307717 | 0.1296752 | 0.4307717 | 0.0008984 | 0.1491266 | 0.0678151 | 0.8264448 | 0.0016209 |
| hsa.miR.140.3p | 3.000561 | 3.018035 | 0.2271711 | 2.949429 | 2.935298 | 0.2005767 | 3.051693 | 3.090265 | 0.2412282 | 0.1022638 | -0.1022638 | -0.3397129 | -0.1022638 | -0.3397129 | 0.0008923 | 0.1481224 | 0.0678151 | 0.8293794 | 0.0016209 |
| hsa.miR.193a.5p | 2.336431 | 2.300347 | 0.3276854 | 2.266929 | 2.278125 | 0.1938504 | 2.405932 | 2.378371 | 0.4104126 | 0.1390021 | -0.1390021 | -0.4617551 | -0.1390021 | -0.4617551 | 0.0018588 | 0.3085666 | 0.1375538 | 0.5106511 | 0.0033179 |
| hsa.let.7b.5p | 4.195831 | 4.209035 | 0.2924159 | 4.256938 | 4.282751 | 0.1889508 | 4.134724 | 4.104211 | 0.3585804 | -0.1222138 | 0.1222138 | 0.4059855 | 0.1222138 | 0.4059855 | 0.0021512 | 0.3570954 | 0.1570359 | 0.4472157 | 0.0037989 |
| hsa.miR.425.5p | 1.923042 | 1.930834 | 0.3243793 | 1.987356 | 1.978949 | 0.2395357 | 1.858729 | 1.880977 | 0.3817182 | -0.1286263 | 0.1286263 | 0.4272874 | 0.1286263 | 0.4272874 | 0.0035774 | 0.5938450 | 0.2575713 | 0.2263269 | 0.0062510 |
| hsa.miR.99b.5p | 4.315202 | 4.300285 | 0.2601134 | 4.263737 | 4.282048 | 0.1951765 | 4.366667 | 4.343604 | 0.3041630 | 0.1029303 | -0.1029303 | -0.3419271 | -0.1029303 | -0.3419271 | 0.0036421 | 0.6045962 | 0.2585923 | 0.2185346 | 0.0062979 |
| hsa.miR.146b.3p | 1.886946 | 1.965307 | 0.4754291 | 1.979342 | 1.993235 | 0.3044790 | 1.794549 | 1.860622 | 0.5867326 | -0.1847929 | 0.1847929 | 0.6138687 | 0.1847929 | 0.6138687 | 0.0043674 | 0.7249845 | 0.3057163 | 0.1396713 | 0.0074741 |
| hsa.miR.128.3p | 1.955925 | 1.943604 | 0.2304255 | 2.000480 | 1.993252 | 0.1994596 | 1.911370 | 1.897153 | 0.2508004 | -0.0891101 | 0.0891101 | 0.2960174 | 0.0891101 | 0.2960174 | 0.0044531 | 0.7392212 | 0.3072666 | 0.1312256 | 0.0075431 |
| hsa.miR.34a.5p | 2.074666 | 2.121651 | 0.2675331 | 2.125712 | 2.136585 | 0.1931050 | 2.023620 | 2.059319 | 0.3182343 | -0.1020923 | 0.1020923 | 0.3391431 | 0.1020923 | 0.3391431 | 0.0050922 | 0.8453067 | 0.3462702 | 0.0729857 | 0.0085385 |
| hsa.miR.1307.5p | 2.032668 | 2.074195 | 0.4746939 | 2.121558 | 2.143052 | 0.3137240 | 1.943777 | 1.930948 | 0.5817384 | -0.1777808 | 0.1777808 | 0.5905749 | 0.1777808 | 0.5905749 | 0.0060341 | 1.0000000 | 0.4042856 | 0.0000000 | 0.0100166 |
| hsa.miR.146a.5p | 1.834930 | 1.878293 | 0.5428638 | 1.936147 | 1.968467 | 0.3763981 | 1.733712 | 1.804289 | 0.6556095 | -0.2024345 | 0.2024345 | 0.6724729 | 0.2024345 | 0.6724729 | 0.0062340 | 1.0000000 | 0.4114442 | 0.0000000 | 0.0102460 |
| hsa.miR.660.5p | 1.637156 | 1.706445 | 0.5648980 | 1.742335 | 1.736172 | 0.1929619 | 1.531977 | 1.656099 | 0.7626676 | -0.2103579 | 0.2103579 | 0.6987938 | 0.2103579 | 0.6987938 | 0.0065781 | 1.0000000 | 0.4275745 | 0.0000000 | 0.0107055 |
| hsa.miR.1307.3p | 2.668366 | 2.682073 | 0.3103661 | 2.725068 | 2.712113 | 0.1957961 | 2.611663 | 2.652759 | 0.3856627 | -0.1134058 | 0.1134058 | 0.3767259 | 0.1134058 | 0.3767259 | 0.0074280 | 1.0000000 | 0.4753900 | 0.0000000 | 0.0119713 |
| hsa.miR.17.3p | 2.280466 | 2.310070 | 0.2760079 | 2.330198 | 2.324300 | 0.1630184 | 2.230733 | 2.251456 | 0.3485844 | -0.0994646 | 0.0994646 | 0.3304142 | 0.0994646 | 0.3304142 | 0.0083333 | 1.0000000 | 0.5249975 | 0.0000000 | 0.0133012 |
| hsa.miR.103a.3p | 4.078473 | 4.104653 | 0.2058716 | 4.115271 | 4.122395 | 0.1680536 | 4.041675 | 4.049734 | 0.2327882 | -0.0735962 | 0.0735962 | 0.2444812 | 0.0735962 | 0.2444812 | 0.0086789 | 1.0000000 | 0.5380907 | 0.0000000 | 0.0137209 |
| hsa.miR.484 | 1.597652 | 1.580193 | 0.2719205 | 1.550795 | 1.553409 | 0.1598979 | 1.644510 | 1.656175 | 0.3443530 | 0.0937157 | -0.0937157 | -0.3113168 | -0.0937157 | -0.3113168 | 0.0116710 | 1.0000000 | 0.7119329 | 0.0000000 | 0.0182773 |
| hsa.miR.452.5p | 1.609759 | 1.684291 | 0.4772755 | 1.691423 | 1.719895 | 0.2895699 | 1.528095 | 1.619481 | 0.6003483 | -0.1633282 | 0.1633282 | 0.5425647 | 0.1633282 | 0.5425647 | 0.0122598 | 1.0000000 | 0.7355852 | 0.0000000 | 0.0189271 |
| hsa.miR.584.5p | 1.562512 | 1.609699 | 0.4055066 | 1.631695 | 1.617142 | 0.3258710 | 1.493330 | 1.570896 | 0.4632148 | -0.1383646 | 0.1383646 | 0.4596372 | 0.1383646 | 0.4596372 | 0.0123140 | 1.0000000 | 0.7355852 | 0.0000000 | 0.0189271 |
| hsa.miR.9.5p | 2.452682 | 2.417320 | 0.5298502 | 2.364790 | 2.362214 | 0.3600516 | 2.540574 | 2.612118 | 0.6472233 | 0.1757844 | -0.1757844 | -0.5839430 | -0.1757844 | -0.5839430 | 0.0151165 | 1.0000000 | 0.8767577 | 0.0000000 | 0.0230215 |
| hsa.miR.361.3p | 2.114070 | 2.138671 | 0.2668813 | 2.157051 | 2.182644 | 0.1719882 | 2.071090 | 2.091288 | 0.3313815 | -0.0859613 | 0.0859613 | 0.2855574 | 0.0859613 | 0.2855574 | 0.0184170 | 1.0000000 | 1.0000000 | 0.0000000 | 0.0277930 |
| hsa.let.7g.5p | 2.743360 | 2.731598 | 0.2633678 | 2.701240 | 2.687710 | 0.2007329 | 2.785480 | 2.824734 | 0.3090374 | 0.0842401 | -0.0842401 | -0.2798397 | -0.0842401 | -0.2798397 | 0.0191012 | 1.0000000 | 1.0000000 | 0.0000000 | 0.0285658 |
| hsa.let.7a.3p | 1.362918 | 1.456311 | 0.5840349 | 1.454096 | 1.472496 | 0.2172292 | 1.271740 | 1.413722 | 0.7883141 | -0.1823566 | 0.1823566 | 0.6057754 | 0.1823566 | 0.6057754 | 0.0227488 | 1.0000000 | 1.0000000 | 0.0000000 | 0.0337170 |
| hsa.miR.28.3p | 3.478699 | 3.452430 | 0.2371295 | 3.442283 | 3.445475 | 0.2164372 | 3.515115 | 3.488616 | 0.2519303 | 0.0728325 | -0.0728325 | -0.2419443 | -0.0728325 | -0.2419443 | 0.0243409 | 1.0000000 | 1.0000000 | 0.0000000 | 0.0355939 |
| hsa.miR.23b.3p | 3.145549 | 3.139008 | 0.1654088 | 3.120156 | 3.106189 | 0.1396735 | 3.170942 | 3.185124 | 0.1848453 | 0.0507862 | -0.0507862 | -0.1687082 | -0.0507862 | -0.1687082 | 0.0244440 | 1.0000000 | 1.0000000 | 0.0000000 | 0.0355939 |
| hsa.miR.29b.3p | 2.730065 | 2.739194 | 0.3702233 | 2.785646 | 2.771371 | 0.2357865 | 2.674484 | 2.713670 | 0.4621597 | -0.1111620 | 0.1111620 | 0.3692721 | 0.1111620 | 0.3692721 | 0.0281053 | 1.0000000 | 1.0000000 | 0.0000000 | 0.0405694 |
We may want to futher visualize the results of differential expression using heatmap and vulcano plot.
Z-scoring the values before clustering and plotting and help to gain more insights.
Let’s plot the vulcano plot and label top 10 most significant miRNAs:
We may also what to check the consistency of differential expression between datasets:
de_test = ks.miRNA_differential_expression(dplyr::select(test, starts_with("hsa")), test$Class)
de_valid = ks.miRNA_differential_expression(dplyr::select(valid, starts_with("hsa")), valid$Class)
ks.correlation_plot(de$log2FC, de_test$log2FC, "log2FC on training set", "log2FC on test set", "", yx = T)The main feature of this package is the shotgun-like feature selection evaluation of possible miRNA signatures of biological processes. The function can be applied in a straightforward way, e.g.:
But, for largers projects we suggest using the following wrapper wich will perform the feature selection in parallel, significantly reducing computational time. We do not recommend using more than 5 threads, beacuse some of the methods inhereditly use multicore processing.
readLines("Tutorial_miRNAselector.R") %>% paste0(collapse = "\n") %>% cat
#> #options(warn = -1)
#> suppressMessages(library(foreach))
#> suppressMessages(library(doParallel))
#> suppressMessages(library(parallel))
#> suppressMessages(library(doSNOW))
#>
#> m = 1:56 # which methods to check?
#>
#> cl <- makeCluster(5) # 5 threds by default
#> doSNOW::registerDoSNOW(cl)
#> iterations = length(m)
#> pb <- txtProgressBar(max = iterations, style = 3)
#> progress <- function(n) setTxtProgressBar(pb, n)
#> opts <- list(progress = progress)
#> foreach(i = m, .options.snow = opts) %dopar%
#> {
#> suppressMessages(library(miRNAselector))
#> setwd("~/public/Projekty/KS/miRNAselector/vignettes") # change it you to your working directory
#> ks.miRNAselector(m = i, max_iterations = 1, stamp = "tutorial", debug = F, # we set debug to false (to make the package smaller), you may also want to change stamp to something meaningful, max_iterations was set to 1 to recude the computational time.. in real life scenarios it is resonable to use at least 10 iterations.
#> prefer_no_features = 11, # Few methods are filter rather than wrapper methods, thus requires the maximum number of maximum features.
#> conda_path = "/home/konrad/anaconda3/bin/conda", # Methods line WxNet requires usage of python. In setup script we create conda enviorment. Providing conda_path makes it easier to activate env. We prefer this apporach over use_condaenv.
#> timeout = 600) # We don't want to wait eternity in this tutorial, just 10 minutes. Timeout is useful for complicated methods. Depending on your CPU 2 days may be reasonable for larger projects.
#> }
#>
#> stopCluster(cl)
#> #options(warn = 0)Few notes about what is does:
m parameter. The aim of this function is to perform feature selection using multiple methods and to create formulas for benchmarking.temp subfolder.Files created for each method (e.g. for stamp=tutorial and m=1):
formulastutorial-1.RDS - main result file containing the final formula (final set of miRNAs selected by this method).time1-formula.RDS - time taken to compute the resultstutorial1featureselection.log - log file of the processall1-tutorial.rdata - all variables created during feature selection (created if debug=T).Pearls about the methods:
Sig = miRNAs with p-value <0.05 after BH correction (DE using t-test)Fcsig = sig + absolute log2FC filter (included if abs. log2FC>1)Cfs = Correlation-based Feature Selection for Machine Learning (more: https://www.cs.waikato.ac.nz/~mhall/thesis.pdf)Classloop = Classification using different classification algorithms (classifiers) with the embedded feature selection and using the different schemes for the performance validation (more: https://rdrr.io/cran/Biocomb/man/classifier.loop.html)Fcfs = CFS algorithm with forward search (https://rdrr.io/cran/Biocomb/man/select.forward.Corr.html)MDL methods = minimal description length (MDL) discretization algorithm with different a method of feature ranking or feature selection (AUC, SU, CorrSF) (more: https://rdrr.io/cran/Biocomb/man/select.process.html)bounceR = genetic algorithm with componentwise boosting (more: https://www.statworx.com/ch/blog/automated-feature-selection-using-bouncer/)RandomForestRFE = recursive feature elimination using random forest with resampling to assess the performance. (more: https://topepo.github.io/caret/recursive-feature-elimination.html#resampling-and-external-validation)GeneticAlgorithmRF (more: https://topepo.github.io/caret/feature-selection-using-genetic-algorithms.html)SimulatedAnnealing = makes small random changes (i.e. perturbations) to an initial candidate solution (more: https://topepo.github.io/caret/feature-selection-using-simulated-annealing.html)Boruta (more: https://www.jstatsoft.org/article/view/v036i11/v36i11.pdf)spFSR = simultaneous perturbation stochastic approximation (SPSA-FSR) (more: https://arxiv.org/abs/1804.05589)varSelRF = using the out-of-bag error as minimization criterion, carry out variable elimination from random forest, by successively eliminating the least important variables (with importance as returned from random forest). (more: https://www.ncbi.nlm.nih.gov/pubmed/16398926)WxNet = a neural network-based feature selection algorithm for transcriptomic data (more: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6642261/)Step = backward stepwise method of feature selection based on logistic regression (GLM, family = binomial) using AIC criteria (stepAIC) and functions from My.stepwise package (https://cran.r-project.org/web/packages/My.stepwise/index.html)Notes about methods:
– TO DO –
The miRNA.selector functions saves all output files to temp/ directory. As users may want to run multiple selection methods in different configurations we do not recommend using the return of this function in the next steps. Instead, we provide ks.merge_formulas() which conviniently summerizes the results of feature selection. We do:
selected_sets_of_miRNAs = ks.merge_formulas(max_miRNAs = 11) # we filter out sets with more than 11 miRNAs.
selected_sets_of_miRNAs_with_own = ks.merge_formulas(max_miRNAs = 11, add = list(my_own_signature = c("hsa.miR.192.5p", "hsa.let.7g.5p", "hsa.let.7a.5p",
"hsa.let.7d.5p", "hsa.miR.194.5p", "hsa.miR.98.5p", "hsa.let.7f.5p", "hsa.miR.26b.5p"))) # you can also add your own signature (for example selected from literature)Note that:
fcsig and cfs_sig are always retained in the final formulas set (ignoring max_miRNA parameter). This is because those methods are usually used as benchmark comparator in discussing the final set of miRNAs. We find more easy to remove those sets (if needed) from final selection than to look for them in temp files or in featureselection_formulas_all.RDS.*.csv files.featureselection_formulas_all.RDS - contains the formulas for all selection methods, (2) featureselection_formulas_final.RDS - contains methods that yelded in less or equal than max_miRNA miRNAs and fcsig and cfs_sig.Let’s analyze the process of feature selection:
all_sets = readRDS("featureselection_formulas_all.RDS")
length(all_sets) # How many feature selection methods completed in time?
#> [1] 47
final_sets = readRDS("featureselection_formulas_final.RDS")
length(final_sets) # How many feature selection methods completed in time and fulfilled max_miRNA criteria? (remember about fcsig and cfs_sig)
#> [1] 33
featureselection_formulas_final = fread("featureselection_formulas_final.csv")
ks.table(featureselection_formulas_final) # show information about selected formulas| name | formula | ile_miRNA |
|---|---|---|
| cfs_sig | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.338.3p + hsa.miR.199b.5p + hsa.miR.381.3p + hsa.miR.199a.3p + hsa.miR.194.5p + hsa.miR.134.5p + hsa.miR.192.5p + hsa.miR.200a.3p + hsa.miR.181c.3p + hsa.miR.29a.3p + hsa.miR.378a.3p + hsa.miR.144.5p + hsa.miR.335.3p + hsa.miR.126.5p + hsa.let.7c.5p + hsa.miR.378a.5p + hsa.miR.181c.5p + hsa.miR.126.3p + hsa.miR.185.5p + hsa.miR.29c.3p + hsa.miR.151a.5p + hsa.miR.92b.3p + hsa.miR.10b.5p + hsa.miR.195.5p + hsa.miR.101.3p + hsa.miR.484 + hsa.miR.28.3p | 34 |
| fcsig | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.139.5p + hsa.miR.199a.5p + hsa.miR.338.3p + hsa.miR.199b.5p + hsa.miR.127.5p + hsa.miR.381.3p + hsa.miR.127.3p + hsa.miR.199a.3p + hsa.miR.181b.5p + hsa.miR.199b.3p + hsa.miR.194.5p + hsa.miR.134.5p + hsa.miR.30a.5p + hsa.miR.200a.5p + hsa.miR.136.5p + hsa.miR.192.5p + hsa.miR.451a + hsa.miR.200a.3p + hsa.miR.141.3p + hsa.miR.200b.3p + hsa.miR.708.3p + hsa.miR.210.3p + hsa.miR.486.5p + hsa.miR.378a.3p + hsa.miR.379.5p + hsa.miR.141.5p + hsa.miR.21.3p + hsa.miR.337.3p + hsa.miR.144.5p + hsa.miR.429 + hsa.miR.335.3p + hsa.miR.126.5p + hsa.miR.181a.3p + hsa.miR.181a.5p + hsa.miR.378a.5p + hsa.miR.146b.5p + hsa.miR.182.5p + hsa.miR.183.5p + hsa.miR.223.3p + hsa.miR.200c.3p + hsa.miR.10b.5p + hsa.miR.142.3p + hsa.miR.142.5p | 50 |
| my_own_signature | Class ~ hsa.miR.192.5p + hsa.let.7g.5p + hsa.let.7a.5p + hsa.let.7d.5p + hsa.miR.194.5p + hsa.miR.98.5p + hsa.let.7f.5p + hsa.miR.26b.5p | 0 |
| bounceR-stability | Class ~ hsa.miR.500a.3p + hsa.miR.126.3p + hsa.miR.125a.5p + hsa.miR.181a.2.3p + hsa.miR.1307.3p + hsa.miR.25.3p + hsa.miR.200b.3p + hsa.miR.22.3p + hsa.miR.106b.3p + hsa.miR.99a.5p + hsa.miR.452.5p | 11 |
| fcfsSMOTE | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p | 2 |
| fcfsSMOTE_sig | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.136.5p | 4 |
| fwrap | Class ~ hsa.miR.21.5p + hsa.miR.126.5p + hsa.miR.375.3p | 3 |
| fwrap_sig | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.375.3p | 3 |
| AUC_MDL | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.139.5p + hsa.miR.127.5p | 11 |
| SU_MDL | Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30e.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.30a.3p + hsa.miR.10b.5p + hsa.miR.126.5p + hsa.miR.199b.5p + hsa.miR.30c.2.3p + hsa.miR.126.3p | 11 |
| CorrSF_MDL | Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30e.3p + hsa.miR.194.5p + hsa.miR.10b.5p + hsa.miR.199b.5p + hsa.miR.30c.2.3p + hsa.miR.338.3p + hsa.miR.217.5p + hsa.miR.378a.5p + hsa.miR.195.5p | 11 |
| sigtop | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.139.5p + hsa.miR.199a.5p + hsa.miR.338.3p + hsa.miR.199b.5p | 11 |
| sigtopBonf | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.139.5p + hsa.miR.199a.5p + hsa.miR.338.3p + hsa.miR.199b.5p | 11 |
| sigtopHolm | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.139.5p + hsa.miR.199a.5p + hsa.miR.338.3p + hsa.miR.199b.5p | 11 |
| topFC | Class ~ hsa.miR.217.5p + hsa.miR.375.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.199b.5p + hsa.miR.200a.3p + hsa.miR.708.3p + hsa.miR.144.5p + hsa.miR.141.3p | 11 |
| sigtopSMOTE | Class ~ hsa.let.7c.5p + hsa.miR.16.5p + hsa.miR.20a.5p + hsa.miR.21.5p + hsa.miR.21.3p + hsa.miR.22.3p + hsa.miR.23a.3p + hsa.miR.26a.5p + hsa.miR.26b.5p + hsa.miR.27a.3p + hsa.miR.29a.3p | 11 |
| sigtopBonfSMOTE | Class ~ hsa.let.7c.5p + hsa.miR.16.5p + hsa.miR.20a.5p + hsa.miR.21.5p + hsa.miR.21.3p + hsa.miR.22.3p + hsa.miR.23a.3p + hsa.miR.26a.5p + hsa.miR.26b.5p + hsa.miR.27a.3p + hsa.miR.29a.3p | 11 |
| sigtopHolmSMOTE | Class ~ hsa.let.7c.5p + hsa.miR.16.5p + hsa.miR.20a.5p + hsa.miR.21.5p + hsa.miR.21.3p + hsa.miR.22.3p + hsa.miR.23a.3p + hsa.miR.26a.5p + hsa.miR.26b.5p + hsa.miR.27a.3p + hsa.miR.29a.3p | 11 |
| topFCSMOTE | Class ~ hsa.miR.217.5p + hsa.miR.375.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.199b.5p + hsa.miR.144.5p + hsa.miR.30c.2.3p + hsa.miR.200a.3p + hsa.miR.708.3p | 11 |
| AUC_MDLSMOTE | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.132.3p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.200a.5p + hsa.miR.127.5p | 11 |
| SU_MDLSMOTE | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.708.3p + hsa.miR.375.3p + hsa.miR.30c.2.3p + hsa.miR.217.5p + hsa.miR.194.5p + hsa.miR.200a.3p + hsa.miR.132.3p + hsa.miR.199a.5p | 11 |
| CorrSF_MDLSMOTE | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.708.3p + hsa.miR.30a.3p + hsa.miR.136.5p + hsa.miR.132.3p | 6 |
| AUC_MDL_sig | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.139.5p + hsa.miR.127.5p | 11 |
| SU_MDL_sig | Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30e.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.30a.3p + hsa.miR.10b.5p + hsa.miR.126.5p + hsa.miR.199b.5p + hsa.miR.30c.2.3p + hsa.miR.126.3p | 11 |
| CorrSF_MDL_sig | Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30e.3p + hsa.miR.194.5p + hsa.miR.10b.5p + hsa.miR.199b.5p + hsa.miR.30c.2.3p + hsa.miR.338.3p + hsa.miR.217.5p + hsa.miR.378a.5p + hsa.miR.195.5p | 11 |
| AUC_MDLSMOTE_sig | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.132.3p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.127.5p + hsa.miR.139.5p | 11 |
| SU_MDLSMOTE_sig | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.375.3p + hsa.miR.708.3p + hsa.miR.136.5p + hsa.miR.30c.2.3p + hsa.miR.194.5p + hsa.miR.200a.3p + hsa.miR.217.5p + hsa.miR.132.3p | 11 |
| CorrSF_MDLSMOTE_sig | Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30a.3p | 3 |
| RandomForestRFE | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.375.3p + hsa.miR.132.3p + hsa.miR.30c.2.3p + hsa.miR.217.5p + hsa.miR.194.5p + hsa.miR.192.5p + hsa.miR.126.5p | 10 |
| cfsSMOTE | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.132.3p + hsa.miR.136.5p + hsa.miR.30e.3p | 5 |
| cfsSMOTE_sig | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.132.3p + hsa.miR.136.5p | 5 |
| Mystepwise_glm_binomial | Class ~ hsa.miR.21.5p + hsa.miR.130a.3p + hsa.miR.30e.3p + hsa.miR.375.3p + hsa.miR.378a.3p + hsa.miR.146b.5p | 6 |
| Mystepwise_sig_glm_binomial | Class ~ hsa.miR.21.5p + hsa.miR.26b.5p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.378a.3p + hsa.miR.338.3p | 6 |
Note that my_own_signture has 0 miRNAs according to the table. This trick is done to make sure that every sigure added manually will survive filtering.
Summary:
psych::describe(featureselection_formulas_final$ile_miRNA[-which(featureselection_formulas_final$ile_miRNA == 0)]) # Descriptive statistics of how many features where selected in the final set.
#> vars n mean sd median trimmed mad min max range skew kurtosis se
#> X1 1 32 10.81 9.01 11 9.35 0 2 50 48 2.96 9.64 1.59In the next step of looking for the best signature, we perform benchmarking. This tests all the signatures using different data mining methods. Here is the example of benchmark with default parameters:
readLines("Tutorial_benchmark.R") %>% paste0(collapse = "\n") %>% cat
#> suppressMessages(library(miRNAselector))
#> ks.benchmark(search_iters = 5, # 5 random hyperparameter sets will be checked; 5 is set here for speed purposes.. for real projects use more
#> algorithms = c("mlp", "mlpML", "svmRadial", "svmLinear", "rf", "C5.0", "rpart",
#> "rpart2", "ctree"), # default set of methods, note that logistic regression (glm) is always included
#> output_file = paste0("benchmark.csv")) # the main outputAs benchmarking is done the main result file is saved as declared in output_file parameter. This file contains the performance metrics of the selected signiture across different methods of data mining modelling. Let’s take a look:
| V1 | method | SMOTE | miRy | glm_modelname | glm_train_ROCAUC | glm_train_ROCAUC_lower95CI | glm_train_ROCAUC_upper95CI | glm_train_Accuracy | glm_train_Sensitivity | glm_train_Specificity | glm_test_Accuracy | glm_test_Sensitivity | glm_test_Specificity | glm_valid_Accuracy | glm_valid_Sensitivity | glm_valid_Specificity | mlp_modelname | mlp_train_ROCAUC | mlp_train_ROCAUC_lower95CI | mlp_train_ROCAUC_upper95CI | mlp_train_Accuracy | mlp_train_Sensitivity | mlp_train_Specificity | mlp_test_Accuracy | mlp_test_Sensitivity | mlp_test_Specificity | mlp_valid_Accuracy | mlp_valid_Sensitivity | mlp_valid_Specificity | mlpML_modelname | mlpML_train_ROCAUC | mlpML_train_ROCAUC_lower95CI | mlpML_train_ROCAUC_upper95CI | mlpML_train_Accuracy | mlpML_train_Sensitivity | mlpML_train_Specificity | mlpML_test_Accuracy | mlpML_test_Sensitivity | mlpML_test_Specificity | mlpML_valid_Accuracy | mlpML_valid_Sensitivity | mlpML_valid_Specificity | svmRadial_modelname | svmRadial_train_ROCAUC | svmRadial_train_ROCAUC_lower95CI | svmRadial_train_ROCAUC_upper95CI | svmRadial_train_Accuracy | svmRadial_train_Sensitivity | svmRadial_train_Specificity | svmRadial_test_Accuracy | svmRadial_test_Sensitivity | svmRadial_test_Specificity | svmRadial_valid_Accuracy | svmRadial_valid_Sensitivity | svmRadial_valid_Specificity | svmLinear_modelname | svmLinear_train_ROCAUC | svmLinear_train_ROCAUC_lower95CI | svmLinear_train_ROCAUC_upper95CI | svmLinear_train_Accuracy | svmLinear_train_Sensitivity | svmLinear_train_Specificity | svmLinear_test_Accuracy | svmLinear_test_Sensitivity | svmLinear_test_Specificity | svmLinear_valid_Accuracy | svmLinear_valid_Sensitivity | svmLinear_valid_Specificity | rf_modelname | rf_train_ROCAUC | rf_train_ROCAUC_lower95CI | rf_train_ROCAUC_upper95CI | rf_train_Accuracy | rf_train_Sensitivity | rf_train_Specificity | rf_test_Accuracy | rf_test_Sensitivity | rf_test_Specificity | rf_valid_Accuracy | rf_valid_Sensitivity | rf_valid_Specificity | C5.0_modelname | C5.0_train_ROCAUC | C5.0_train_ROCAUC_lower95CI | C5.0_train_ROCAUC_upper95CI | C5.0_train_Accuracy | C5.0_train_Sensitivity | C5.0_train_Specificity | C5.0_test_Accuracy | C5.0_test_Sensitivity | C5.0_test_Specificity | C5.0_valid_Accuracy | C5.0_valid_Sensitivity | C5.0_valid_Specificity | rpart_modelname | rpart_train_ROCAUC | rpart_train_ROCAUC_lower95CI | rpart_train_ROCAUC_upper95CI | rpart_train_Accuracy | rpart_train_Sensitivity | rpart_train_Specificity | rpart_test_Accuracy | rpart_test_Sensitivity | rpart_test_Specificity | rpart_valid_Accuracy | rpart_valid_Sensitivity | rpart_valid_Specificity | rpart2_modelname | rpart2_train_ROCAUC | rpart2_train_ROCAUC_lower95CI | rpart2_train_ROCAUC_upper95CI | rpart2_train_Accuracy | rpart2_train_Sensitivity | rpart2_train_Specificity | rpart2_test_Accuracy | rpart2_test_Sensitivity | rpart2_test_Specificity | rpart2_valid_Accuracy | rpart2_valid_Sensitivity | rpart2_valid_Specificity | ctree_modelname | ctree_train_ROCAUC | ctree_train_ROCAUC_lower95CI | ctree_train_ROCAUC_upper95CI | ctree_train_Accuracy | ctree_train_Sensitivity | ctree_train_Specificity | ctree_test_Accuracy | ctree_test_Sensitivity | ctree_test_Specificity | ctree_valid_Accuracy | ctree_valid_Sensitivity | ctree_valid_Specificity |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | cfs_sig | No | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.338.3p + hsa.miR.199b.5p + hsa.miR.381.3p + hsa.miR.199a.3p + hsa.miR.194.5p + hsa.miR.134.5p + hsa.miR.192.5p + hsa.miR.200a.3p + hsa.miR.181c.3p + hsa.miR.29a.3p + hsa.miR.378a.3p + hsa.miR.144.5p + hsa.miR.335.3p + hsa.miR.126.5p + hsa.let.7c.5p + hsa.miR.378a.5p + hsa.miR.181c.5p + hsa.miR.126.3p + hsa.miR.185.5p + hsa.miR.29c.3p + hsa.miR.151a.5p + hsa.miR.92b.3p + hsa.miR.10b.5p + hsa.miR.195.5p + hsa.miR.101.3p + hsa.miR.484 + hsa.miR.28.3p | 1584462908 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.8888889 | 0.9166667 | 0.8611111 | 1.0000000 | 1.0000000 | 1.0000000 | 1584463328 | 0.9974670 | 0.9934914 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9722222 | 1.0000000 | 0.9444444 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463844 | 0.9963316 | 0.9910107 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9722222 | 1.0000000 | 0.9444444 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464368 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9583333 | 0.9722222 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465010 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9583333 | 0.9722222 | 0.9444444 | 0.9714286 | 1.0000000 | 0.9428571 | 1584465465 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584466016 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9861111 | 1.0000000 | 0.9722222 | 0.9857143 | 1.0000000 | 0.9714286 | 1584467102 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467489 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467877 | 0.9768539 | 0.9567499 | 0.9969579 | 0.9485981 | 0.9813084 | 0.9158879 | 0.9305556 | 1.0000000 | 0.8611111 | 0.9571429 | 1.0000000 | 0.9142857 |
| 2 | fcsig | No | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.139.5p + hsa.miR.199a.5p + hsa.miR.338.3p + hsa.miR.199b.5p + hsa.miR.127.5p + hsa.miR.381.3p + hsa.miR.127.3p + hsa.miR.199a.3p + hsa.miR.181b.5p + hsa.miR.199b.3p + hsa.miR.194.5p + hsa.miR.134.5p + hsa.miR.30a.5p + hsa.miR.200a.5p + hsa.miR.136.5p + hsa.miR.192.5p + hsa.miR.451a + hsa.miR.200a.3p + hsa.miR.141.3p + hsa.miR.200b.3p + hsa.miR.708.3p + hsa.miR.210.3p + hsa.miR.486.5p + hsa.miR.378a.3p + hsa.miR.379.5p + hsa.miR.141.5p + hsa.miR.21.3p + hsa.miR.337.3p + hsa.miR.144.5p + hsa.miR.429 + hsa.miR.335.3p + hsa.miR.126.5p + hsa.miR.181a.3p + hsa.miR.181a.5p + hsa.miR.378a.5p + hsa.miR.146b.5p + hsa.miR.182.5p + hsa.miR.183.5p + hsa.miR.223.3p + hsa.miR.200c.3p + hsa.miR.10b.5p + hsa.miR.142.3p + hsa.miR.142.5p | 1584462916 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9166667 | 0.9166667 | 0.9166667 | 0.9571429 | 1.0000000 | 0.9142857 | 1584463336 | 0.5373832 | 0.5123484 | 0.5624180 | 0.5000000 | 0.0000000 | 1.0000000 | 0.5000000 | 0.0000000 | 1.0000000 | 0.5000000 | 0.0000000 | 1.0000000 | 1584463852 | 0.9979037 | 0.9948019 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9722222 | 1.0000000 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584464376 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9722222 | 1.0000000 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465017 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9305556 | 1.0000000 | 0.8611111 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465473 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584466026 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9861111 | 1.0000000 | 0.9722222 | 0.9857143 | 1.0000000 | 0.9714286 | 1584467111 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467497 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467885 | 0.9943663 | 0.9892504 | 0.9994823 | 0.9579439 | 0.9345794 | 0.9813084 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9571429 | 0.9714286 | 0.9428571 |
| 3 | my_own_signature | No | Class ~ hsa.miR.192.5p + hsa.let.7g.5p + hsa.let.7a.5p + hsa.let.7d.5p + hsa.miR.194.5p + hsa.miR.98.5p + hsa.let.7f.5p + hsa.miR.26b.5p | 1584462924 | 0.9239235 | 0.8901386 | 0.9577083 | 0.8317757 | 0.8504673 | 0.8130841 | 0.8472222 | 0.8055556 | 0.8888889 | 0.8857143 | 0.8857143 | 0.8857143 | 1584463344 | 0.9246222 | 0.8917772 | 0.9574673 | 0.8130841 | 0.9813084 | 0.6448598 | 0.7638889 | 0.9722222 | 0.5555556 | 0.8428571 | 1.0000000 | 0.6857143 | 1584463860 | 0.9363263 | 0.9041742 | 0.9684784 | 0.8411215 | 0.9626168 | 0.7196262 | 0.8055556 | 0.9722222 | 0.6388889 | 0.8857143 | 1.0000000 | 0.7714286 | 1584464384 | 0.9765918 | 0.9565725 | 0.9966111 | 0.9299065 | 0.9719626 | 0.8878505 | 0.8750000 | 0.8888889 | 0.8611111 | 0.9714286 | 0.9714286 | 0.9714286 | 1584465025 | 0.9195563 | 0.8844029 | 0.9547097 | 0.8317757 | 0.8504673 | 0.8130841 | 0.8194444 | 0.8055556 | 0.8333333 | 0.8571429 | 0.8857143 | 0.8285714 | 1584465481 | 1 | 1 | 1 | 1 | 1 | 1 | 0.8611111 | 0.8611111 | 0.8611111 | 0.9428571 | 0.9714286 | 0.9142857 | 1584466034 | 0.9985152 | 0.9963905 | 1.0000000 | 0.9719626 | 0.9813084 | 0.9626168 | 0.8472222 | 0.8055556 | 0.8888889 | 0.9142857 | 0.8857143 | 0.9428571 | 1584467119 | 0.9508254 | 0.9228791 | 0.9787717 | 0.9018692 | 0.8504673 | 0.9532710 | 0.7638889 | 0.6944444 | 0.8333333 | 0.8571429 | 0.7714286 | 0.9428571 | 1584467505 | 0.9021312 | 0.8620675 | 0.9421949 | 0.8831776 | 0.9065421 | 0.8598131 | 0.7500000 | 0.7222222 | 0.7777778 | 0.9285714 | 0.9428571 | 0.9142857 | 1584467893 | 0.9753254 | 0.9605664 | 0.9900843 | 0.8971963 | 0.8317757 | 0.9626168 | 0.7916667 | 0.6944444 | 0.8888889 | 0.8714286 | 0.8000000 | 0.9428571 |
| 4 | bounceR.stability | No | Class ~ hsa.miR.500a.3p + hsa.miR.126.3p + hsa.miR.125a.5p + hsa.miR.181a.2.3p + hsa.miR.1307.3p + hsa.miR.25.3p + hsa.miR.200b.3p + hsa.miR.22.3p + hsa.miR.106b.3p + hsa.miR.99a.5p + hsa.miR.452.5p | 1584462932 | 0.9645384 | 0.9421954 | 0.9868814 | 0.9018692 | 0.9158879 | 0.8878505 | 0.8472222 | 0.8333333 | 0.8611111 | 0.9428571 | 1.0000000 | 0.8857143 | 1584463352 | 0.9375491 | 0.9027342 | 0.9723640 | 0.8738318 | 0.9626168 | 0.7850467 | 0.9027778 | 0.9722222 | 0.8333333 | 0.9142857 | 1.0000000 | 0.8285714 | 1584463868 | 0.9289021 | 0.8945307 | 0.9632734 | 0.8457944 | 0.8317757 | 0.8598131 | 0.8611111 | 0.8333333 | 0.8888889 | 0.9000000 | 0.9142857 | 0.8857143 | 1584464392 | 0.9996506 | 0.9988872 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584465033 | 0.9626168 | 0.9396477 | 0.9855859 | 0.8971963 | 0.9158879 | 0.8785047 | 0.8750000 | 0.9166667 | 0.8333333 | 0.9428571 | 0.9714286 | 0.9142857 | 1584465489 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584466043 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.8750000 | 0.8888889 | 0.8611111 | 0.9571429 | 0.9714286 | 0.9428571 | 1584467127 | 0.9646257 | 0.9402313 | 0.9890202 | 0.9392523 | 0.9532710 | 0.9252336 | 0.8611111 | 0.8888889 | 0.8333333 | 0.9142857 | 0.9714286 | 0.8571429 | 1584467513 | 0.9646257 | 0.9402313 | 0.9890202 | 0.9392523 | 0.9532710 | 0.9252336 | 0.8611111 | 0.8888889 | 0.8333333 | 0.9142857 | 0.9714286 | 0.8571429 | 1584467901 | 0.9828806 | 0.9706308 | 0.9951304 | 0.9252336 | 0.9345794 | 0.9158879 | 0.8611111 | 0.8888889 | 0.8333333 | 0.9142857 | 1.0000000 | 0.8285714 |
| 5 | fcfsSMOTE | Yes | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p | 1584462940 | 0.9907048 | 0.9895335 | 0.9918762 | 0.9694983 | 0.9733645 | 0.9656704 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463372 | 0.9904611 | 0.9892241 | 0.9916980 | 0.9768447 | 0.9824299 | 0.9713149 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463887 | 0.9904518 | 0.9892245 | 0.9916792 | 0.9642907 | 0.9541121 | 0.9743685 | 0.9027778 | 0.9166667 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464430 | 0.9899233 | 0.9882711 | 0.9915755 | 0.9866090 | 1.0000000 | 0.9733506 | 0.8750000 | 0.8888889 | 0.8611111 | 0.9285714 | 0.9428571 | 0.9142857 | 1584465048 | 0.9906759 | 0.9895011 | 0.9918506 | 0.9738690 | 0.9819626 | 0.9658555 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584465506 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9027778 | 0.8055556 | 1.0000000 | 0.8285714 | 0.6857143 | 0.9714286 | 1584466097 | 0.9989302 | 0.9986219 | 0.9992384 | 0.9969312 | 1.0000000 | 0.9938928 | 0.9305556 | 0.8888889 | 0.9722222 | 0.9000000 | 0.8285714 | 0.9714286 | 1584467136 | 0.9729217 | 0.9708267 | 0.9750166 | 0.9712652 | 0.9913084 | 0.9514204 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584467522 | 0.9729217 | 0.9708267 | 0.9750166 | 0.9712652 | 0.9913084 | 0.9514204 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584467909 | 0.9993709 | 0.9991560 | 0.9995858 | 0.9973497 | 1.0000000 | 0.9947256 | 0.9166667 | 0.8611111 | 0.9722222 | 0.9571429 | 0.9428571 | 0.9714286 |
| 6 | fcfsSMOTE_sig | Yes | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.136.5p | 1584462957 | 0.9965117 | 0.9958363 | 0.9971871 | 0.9800995 | 0.9822430 | 0.9779772 | 0.9583333 | 0.9722222 | 0.9444444 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463397 | 0.9953074 | 0.9944878 | 0.9961270 | 0.9842842 | 0.9822430 | 0.9863052 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463912 | 0.9955242 | 0.9947351 | 0.9963132 | 0.9790766 | 0.9913084 | 0.9669659 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9571429 | 1.0000000 | 0.9142857 | 1584464464 | 0.9977802 | 0.9971010 | 0.9984594 | 0.9940020 | 1.0000000 | 0.9880633 | 0.9305556 | 0.9166667 | 0.9444444 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465068 | 0.9958849 | 0.9951245 | 0.9966454 | 0.9841447 | 0.9822430 | 0.9860276 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 1.0000000 | 0.9428571 | 1584465529 | 1 | 1 | 1 | 1 | 1 | 1 | 0.8888889 | 0.8333333 | 0.9444444 | 0.9571429 | 0.9142857 | 1.0000000 | 1584466155 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9995815 | 1.0000000 | 0.9991672 | 0.9305556 | 0.9166667 | 0.9444444 | 0.9571429 | 0.9428571 | 0.9714286 | 1584467150 | 0.9729217 | 0.9708267 | 0.9750166 | 0.9712652 | 0.9913084 | 0.9514204 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584467536 | 0.9866442 | 0.9851528 | 0.9881355 | 0.9858651 | 0.9913084 | 0.9804756 | 0.9305556 | 0.9444444 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584467926 | 0.9996475 | 0.9994508 | 0.9998442 | 0.9980936 | 1.0000000 | 0.9962062 | 0.9305556 | 0.9166667 | 0.9444444 | 0.9285714 | 0.8857143 | 0.9714286 |
| 7 | fwrap | No | Class ~ hsa.miR.21.5p + hsa.miR.126.5p + hsa.miR.375.3p | 1584462973 | 0.9888200 | 0.9789249 | 0.9987150 | 0.9485981 | 0.9439252 | 0.9532710 | 0.9027778 | 0.9444444 | 0.8611111 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463409 | 0.9765918 | 0.9608023 | 0.9923814 | 0.9252336 | 0.9626168 | 0.8878505 | 0.8333333 | 0.9722222 | 0.6944444 | 0.9428571 | 0.9714286 | 0.9142857 | 1584463924 | 0.9791248 | 0.9645951 | 0.9936545 | 0.9252336 | 0.9719626 | 0.8785047 | 0.8194444 | 0.9722222 | 0.6666667 | 0.9142857 | 1.0000000 | 0.8285714 | 1584464481 | 0.9902175 | 0.9749209 | 1.0000000 | 0.9813084 | 1.0000000 | 0.9626168 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465083 | 0.9886453 | 0.9786723 | 0.9986183 | 0.9485981 | 0.9439252 | 0.9532710 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465541 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584466167 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584467164 | 0.9863307 | 0.9714585 | 1.0000000 | 0.9766355 | 0.9813084 | 0.9719626 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584467550 | 0.9863307 | 0.9714585 | 1.0000000 | 0.9766355 | 0.9813084 | 0.9719626 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584467942 | 0.9863307 | 0.9714585 | 1.0000000 | 0.9766355 | 0.9813084 | 0.9719626 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 1.0000000 | 0.9428571 |
| 8 | fwrap_sig | No | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.375.3p | 1584462981 | 0.9924011 | 0.9795847 | 1.0000000 | 0.9859813 | 0.9906542 | 0.9813084 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463417 | 0.9924011 | 0.9800430 | 1.0000000 | 0.9766355 | 0.9906542 | 0.9626168 | 0.9166667 | 0.9722222 | 0.8611111 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463932 | 0.9924011 | 0.9800430 | 1.0000000 | 0.9859813 | 0.9906542 | 0.9813084 | 0.9027778 | 0.9444444 | 0.8611111 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464489 | 0.9956328 | 0.9883696 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465091 | 0.9925758 | 0.9797890 | 1.0000000 | 0.9813084 | 0.9813084 | 0.9813084 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465549 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584466174 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584467172 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467558 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467950 | 0.9906979 | 0.9797015 | 1.0000000 | 0.9579439 | 0.9345794 | 0.9813084 | 0.9166667 | 0.9166667 | 0.9166667 | 0.9428571 | 0.9714286 | 0.9142857 |
| 9 | AUC_MDL | No | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.139.5p + hsa.miR.127.5p | 1584462988 | 0.9931872 | 0.9810931 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463425 | 0.9904795 | 0.9759992 | 1.0000000 | 0.9766355 | 0.9719626 | 0.9813084 | 0.9722222 | 1.0000000 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584463940 | 0.9897808 | 0.9746001 | 1.0000000 | 0.9859813 | 0.9906542 | 0.9813084 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464498 | 0.9910036 | 0.9740380 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9583333 | 0.9722222 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465098 | 0.9894314 | 0.9729248 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9444444 | 0.9722222 | 0.9166667 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465557 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9722222 | 1.0000000 | 0.9444444 | 0.9857143 | 1.0000000 | 0.9714286 | 1584466183 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584467180 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467566 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467958 | 0.9941480 | 0.9887442 | 0.9995518 | 0.9579439 | 0.9345794 | 0.9813084 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9571429 | 0.9714286 | 0.9428571 |
| 10 | SU_MDL | No | Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30e.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.30a.3p + hsa.miR.10b.5p + hsa.miR.126.5p + hsa.miR.199b.5p + hsa.miR.30c.2.3p + hsa.miR.126.3p | 1584462996 | 0.9965936 | 0.9915862 | 1.0000000 | 0.9766355 | 0.9813084 | 0.9719626 | 0.9305556 | 0.9444444 | 0.9166667 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463433 | 0.9927505 | 0.9813321 | 1.0000000 | 0.9813084 | 0.9906542 | 0.9719626 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463948 | 0.9930125 | 0.9822433 | 1.0000000 | 0.9813084 | 0.9906542 | 0.9719626 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584464506 | 0.9969430 | 0.9910570 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9722222 | 1.0000000 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465106 | 0.9937986 | 0.9837316 | 1.0000000 | 0.9813084 | 0.9906542 | 0.9719626 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 1.0000000 | 0.9428571 | 1584465565 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584466190 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9722222 | 1.0000000 | 0.9444444 | 0.9571429 | 0.9714286 | 0.9428571 | 1584467188 | 0.9889073 | 0.9755302 | 1.0000000 | 0.9672897 | 0.9439252 | 0.9906542 | 0.9305556 | 0.9444444 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467573 | 0.9889073 | 0.9755302 | 1.0000000 | 0.9672897 | 0.9439252 | 0.9906542 | 0.9305556 | 0.9444444 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467965 | 0.9825749 | 0.9652791 | 0.9998707 | 0.9579439 | 0.9345794 | 0.9813084 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9571429 | 0.9714286 | 0.9428571 |
| 11 | CorrSF_MDL | No | Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30e.3p + hsa.miR.194.5p + hsa.miR.10b.5p + hsa.miR.199b.5p + hsa.miR.30c.2.3p + hsa.miR.338.3p + hsa.miR.217.5p + hsa.miR.378a.5p + hsa.miR.195.5p | 1584463004 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463441 | 0.9966809 | 0.9911475 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463956 | 0.9964189 | 0.9905142 | 1.0000000 | 0.9766355 | 0.9719626 | 0.9813084 | 0.9583333 | 0.9722222 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584464514 | 0.9998253 | 0.9994066 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9444444 | 0.9722222 | 0.9166667 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465114 | 0.9976417 | 0.9931160 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465573 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584466198 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9722222 | 1.0000000 | 0.9444444 | 0.9571429 | 0.9714286 | 0.9428571 | 1584467196 | 0.9889073 | 0.9755302 | 1.0000000 | 0.9672897 | 0.9439252 | 0.9906542 | 0.9166667 | 0.9166667 | 0.9166667 | 0.9571429 | 0.9714286 | 0.9428571 | 1584467581 | 0.9889073 | 0.9755302 | 1.0000000 | 0.9672897 | 0.9439252 | 0.9906542 | 0.9166667 | 0.9166667 | 0.9166667 | 0.9571429 | 0.9714286 | 0.9428571 | 1584467973 | 0.9942790 | 0.9890146 | 0.9995433 | 0.9579439 | 0.9345794 | 0.9813084 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9571429 | 0.9714286 | 0.9428571 |
| 12 | sigtop | No | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.139.5p + hsa.miR.199a.5p + hsa.miR.338.3p + hsa.miR.199b.5p | 1584463012 | 0.9961569 | 0.9902191 | 1.0000000 | 0.9859813 | 1.0000000 | 0.9719626 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463449 | 0.9912219 | 0.9784013 | 1.0000000 | 0.9813084 | 0.9813084 | 0.9813084 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463964 | 0.9929251 | 0.9817767 | 1.0000000 | 0.9813084 | 0.9813084 | 0.9813084 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464522 | 0.9939733 | 0.9822990 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9722222 | 1.0000000 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465122 | 0.9926631 | 0.9806675 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465580 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584466206 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584467204 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467589 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467981 | 0.9905232 | 0.9794706 | 1.0000000 | 0.9579439 | 0.9345794 | 0.9813084 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9571429 | 0.9714286 | 0.9428571 |
| 13 | sigtopBonf | No | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.139.5p + hsa.miR.199a.5p + hsa.miR.338.3p + hsa.miR.199b.5p | 1584463020 | 0.9961569 | 0.9902191 | 1.0000000 | 0.9859813 | 1.0000000 | 0.9719626 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463456 | 0.9913530 | 0.9788234 | 1.0000000 | 0.9719626 | 0.9626168 | 0.9813084 | 0.9722222 | 1.0000000 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584463972 | 0.9903922 | 0.9759600 | 1.0000000 | 0.9813084 | 1.0000000 | 0.9626168 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464530 | 0.9942353 | 0.9830720 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465129 | 0.9926631 | 0.9806675 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465588 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584466215 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584467212 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467597 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467989 | 0.9905232 | 0.9794706 | 1.0000000 | 0.9579439 | 0.9345794 | 0.9813084 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9571429 | 0.9714286 | 0.9428571 |
| 14 | sigtopHolm | No | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.139.5p + hsa.miR.199a.5p + hsa.miR.338.3p + hsa.miR.199b.5p | 1584463027 | 0.9961569 | 0.9902191 | 1.0000000 | 0.9859813 | 1.0000000 | 0.9719626 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463464 | 0.9917023 | 0.9792337 | 1.0000000 | 0.9719626 | 0.9626168 | 0.9813084 | 0.9722222 | 1.0000000 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584463979 | 0.9914403 | 0.9788481 | 1.0000000 | 0.9813084 | 0.9813084 | 0.9813084 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464538 | 0.9939733 | 0.9822990 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9722222 | 1.0000000 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465137 | 0.9926631 | 0.9806675 | 1.0000000 | 0.9859813 | 0.9906542 | 0.9813084 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465596 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584466223 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584467220 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467605 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467996 | 0.9905232 | 0.9794706 | 1.0000000 | 0.9579439 | 0.9345794 | 0.9813084 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9571429 | 0.9714286 | 0.9428571 |
| 15 | topFC | No | Class ~ hsa.miR.217.5p + hsa.miR.375.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.199b.5p + hsa.miR.200a.3p + hsa.miR.708.3p + hsa.miR.144.5p + hsa.miR.141.3p | 1584463035 | 0.9939733 | 0.9856621 | 1.0000000 | 0.9813084 | 0.9906542 | 0.9719626 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463472 | 0.9910909 | 0.9784150 | 1.0000000 | 0.9859813 | 1.0000000 | 0.9719626 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463987 | 0.9916150 | 0.9800129 | 1.0000000 | 0.9859813 | 1.0000000 | 0.9719626 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464546 | 0.9942353 | 0.9868483 | 1.0000000 | 0.9766355 | 0.9813084 | 0.9719626 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465145 | 0.9939733 | 0.9853614 | 1.0000000 | 0.9813084 | 0.9906542 | 0.9719626 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465604 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584466230 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9714286 | 1.0000000 | 0.9428571 | 1584467228 | 0.9816578 | 0.9643889 | 0.9989267 | 0.9672897 | 0.9719626 | 0.9626168 | 0.9583333 | 0.9722222 | 0.9444444 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467613 | 0.9654118 | 0.9421733 | 0.9886504 | 0.9626168 | 1.0000000 | 0.9252336 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584468005 | 0.9944537 | 0.9893380 | 0.9995693 | 0.9579439 | 0.9906542 | 0.9252336 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 |
| 16 | sigtopSMOTE | Yes | Class ~ hsa.let.7c.5p + hsa.miR.16.5p + hsa.miR.20a.5p + hsa.miR.21.5p + hsa.miR.21.3p + hsa.miR.22.3p + hsa.miR.23a.3p + hsa.miR.26a.5p + hsa.miR.26b.5p + hsa.miR.27a.3p + hsa.miR.29a.3p | 1584463045 | 0.9956204 | 0.9951104 | 0.9961305 | 0.9787976 | 0.9814953 | 0.9761266 | 0.8888889 | 0.9166667 | 0.8611111 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463494 | 0.9986075 | 0.9983098 | 0.9989052 | 0.9928860 | 1.0000000 | 0.9858425 | 0.8888889 | 0.8888889 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584464009 | 0.9976486 | 0.9971144 | 0.9981827 | 0.9883759 | 0.9908411 | 0.9859350 | 0.8750000 | 0.8888889 | 0.8611111 | 0.9714286 | 1.0000000 | 0.9428571 | 1584464572 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9981866 | 1.0000000 | 0.9963912 | 0.9166667 | 0.8611111 | 0.9722222 | 0.9857143 | 0.9714286 | 1.0000000 | 1584465165 | 0.9948269 | 0.9941933 | 0.9954605 | 0.9790301 | 0.9814953 | 0.9765892 | 0.8888889 | 0.9166667 | 0.8611111 | 0.9714286 | 1.0000000 | 0.9428571 | 1584465634 | 1 | 1 | 1 | 1 | 1 | 1 | 0.8472222 | 0.7222222 | 0.9722222 | 0.9571429 | 0.9142857 | 1.0000000 | 1584466293 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9999535 | 1.0000000 | 0.9999075 | 0.9166667 | 0.8611111 | 0.9722222 | 0.9857143 | 0.9714286 | 1.0000000 | 1584467238 | 0.9779993 | 0.9760129 | 0.9799858 | 0.9682429 | 0.9816822 | 0.9549366 | 0.8750000 | 0.8333333 | 0.9166667 | 0.9285714 | 0.9142857 | 0.9428571 | 1584467623 | 0.9393083 | 0.9361198 | 0.9424969 | 0.9393686 | 0.9271963 | 0.9514204 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584468013 | 0.9999095 | 0.9998116 | 1.0000000 | 0.9993026 | 1.0000000 | 0.9986120 | 0.8888889 | 0.8055556 | 0.9722222 | 0.9571429 | 0.9142857 | 1.0000000 |
| 17 | sigtopBonfSMOTE | Yes | Class ~ hsa.let.7c.5p + hsa.miR.16.5p + hsa.miR.20a.5p + hsa.miR.21.5p + hsa.miR.21.3p + hsa.miR.22.3p + hsa.miR.23a.3p + hsa.miR.26a.5p + hsa.miR.26b.5p + hsa.miR.27a.3p + hsa.miR.29a.3p | 1584463064 | 0.9956204 | 0.9951104 | 0.9961305 | 0.9787976 | 0.9814953 | 0.9761266 | 0.8888889 | 0.9166667 | 0.8611111 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463520 | 0.9990513 | 0.9988740 | 0.9992287 | 0.9924211 | 1.0000000 | 0.9849172 | 0.8750000 | 0.8888889 | 0.8611111 | 0.9714286 | 1.0000000 | 0.9428571 | 1584464035 | 0.9985652 | 0.9982756 | 0.9988547 | 0.9917236 | 1.0000000 | 0.9835292 | 0.8888889 | 0.8888889 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584464606 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9983261 | 1.0000000 | 0.9966688 | 0.9027778 | 0.8333333 | 0.9722222 | 0.9857143 | 0.9714286 | 1.0000000 | 1584465191 | 0.9948269 | 0.9941933 | 0.9954605 | 0.9789371 | 0.9814953 | 0.9764042 | 0.8888889 | 0.9166667 | 0.8611111 | 0.9714286 | 1.0000000 | 0.9428571 | 1584465666 | 1 | 1 | 1 | 1 | 1 | 1 | 0.8472222 | 0.7222222 | 0.9722222 | 0.9571429 | 0.9142857 | 1.0000000 | 1584466362 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9999535 | 1.0000000 | 0.9999075 | 0.9166667 | 0.8611111 | 0.9722222 | 0.9857143 | 0.9714286 | 1.0000000 | 1584467254 | 0.9779993 | 0.9760129 | 0.9799858 | 0.9682429 | 0.9816822 | 0.9549366 | 0.8750000 | 0.8333333 | 0.9166667 | 0.9285714 | 0.9142857 | 0.9428571 | 1584467640 | 0.9393083 | 0.9361198 | 0.9424969 | 0.9393686 | 0.9271963 | 0.9514204 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584468032 | 0.9999095 | 0.9998116 | 1.0000000 | 0.9993026 | 1.0000000 | 0.9986120 | 0.8888889 | 0.8055556 | 0.9722222 | 0.9571429 | 0.9142857 | 1.0000000 |
| 18 | sigtopHolmSMOTE | Yes | Class ~ hsa.let.7c.5p + hsa.miR.16.5p + hsa.miR.20a.5p + hsa.miR.21.5p + hsa.miR.21.3p + hsa.miR.22.3p + hsa.miR.23a.3p + hsa.miR.26a.5p + hsa.miR.26b.5p + hsa.miR.27a.3p + hsa.miR.29a.3p | 1584463084 | 0.9956204 | 0.9951104 | 0.9961305 | 0.9787976 | 0.9814953 | 0.9761266 | 0.8888889 | 0.9166667 | 0.8611111 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463545 | 0.9978832 | 0.9975004 | 0.9982660 | 0.9884224 | 0.9908411 | 0.9860276 | 0.8750000 | 0.8888889 | 0.8611111 | 0.9571429 | 1.0000000 | 0.9142857 | 1584464061 | 0.9981969 | 0.9978385 | 0.9985553 | 0.9920491 | 1.0000000 | 0.9841769 | 0.8888889 | 0.8888889 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584464639 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.8750000 | 0.7777778 | 0.9722222 | 0.9714286 | 0.9428571 | 1.0000000 | 1584465217 | 0.9948269 | 0.9941933 | 0.9954605 | 0.9790301 | 0.9814953 | 0.9765892 | 0.8888889 | 0.9166667 | 0.8611111 | 0.9714286 | 1.0000000 | 0.9428571 | 1584465698 | 1 | 1 | 1 | 1 | 1 | 1 | 0.8611111 | 0.7500000 | 0.9722222 | 0.9571429 | 0.9142857 | 1.0000000 | 1584466430 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9999535 | 1.0000000 | 0.9999075 | 0.9166667 | 0.8611111 | 0.9722222 | 0.9857143 | 0.9714286 | 1.0000000 | 1584467271 | 0.9779993 | 0.9760129 | 0.9799858 | 0.9682429 | 0.9816822 | 0.9549366 | 0.8750000 | 0.8333333 | 0.9166667 | 0.9285714 | 0.9142857 | 0.9428571 | 1584467656 | 0.9393083 | 0.9361198 | 0.9424969 | 0.9393686 | 0.9271963 | 0.9514204 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584468051 | 0.9999095 | 0.9998116 | 1.0000000 | 0.9993026 | 1.0000000 | 0.9986120 | 0.8888889 | 0.8055556 | 0.9722222 | 0.9571429 | 0.9142857 | 1.0000000 |
| 19 | topFCSMOTE | Yes | Class ~ hsa.miR.217.5p + hsa.miR.375.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.199b.5p + hsa.miR.144.5p + hsa.miR.30c.2.3p + hsa.miR.200a.3p + hsa.miR.708.3p | 1584463103 | 0.9978276 | 0.9973654 | 0.9982897 | 0.9933045 | 1.0000000 | 0.9866753 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463571 | 0.9991518 | 0.9989223 | 0.9993814 | 0.9926536 | 0.9907477 | 0.9945406 | 0.9305556 | 0.9722222 | 0.8888889 | 1.0000000 | 1.0000000 | 1.0000000 | 1584464087 | 0.9991941 | 0.9989774 | 0.9994108 | 0.9969312 | 1.0000000 | 0.9938928 | 0.9305556 | 0.9722222 | 0.8888889 | 1.0000000 | 1.0000000 | 1.0000000 | 1584464669 | 0.9998316 | 0.9997065 | 0.9999567 | 0.9994420 | 1.0000000 | 0.9988896 | 0.9305556 | 0.9166667 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465236 | 0.9976015 | 0.9971146 | 0.9980884 | 0.9935370 | 1.0000000 | 0.9871380 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465728 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9722222 | 1.0000000 | 0.9444444 | 0.9714286 | 0.9428571 | 1.0000000 | 1584466498 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9722222 | 1.0000000 | 0.9444444 | 0.9857143 | 1.0000000 | 0.9714286 | 1584467287 | 0.9920982 | 0.9909715 | 0.9932249 | 0.9913517 | 1.0000000 | 0.9827889 | 0.9722222 | 1.0000000 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584467673 | 0.9918549 | 0.9907060 | 0.9930039 | 0.9868880 | 0.9910280 | 0.9827889 | 0.9722222 | 1.0000000 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584468070 | 0.9999987 | 0.9999973 | 1.0000000 | 0.9998605 | 1.0000000 | 0.9997224 | 0.9166667 | 0.8888889 | 0.9444444 | 0.9571429 | 0.9142857 | 1.0000000 |
| 20 | AUC_MDLSMOTE | Yes | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.132.3p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.200a.5p + hsa.miR.127.5p | 1584463123 | 0.9986041 | 0.9982479 | 0.9989603 | 0.9944669 | 1.0000000 | 0.9889886 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463596 | 0.9962802 | 0.9954578 | 0.9971026 | 0.9935370 | 1.0000000 | 0.9871380 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464114 | 0.9979454 | 0.9972658 | 0.9986251 | 0.9954433 | 1.0000000 | 0.9909318 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 1.0000000 | 0.9428571 | 1584464699 | 0.9997539 | 0.9996008 | 0.9999069 | 0.9994420 | 1.0000000 | 0.9988896 | 0.9166667 | 0.8611111 | 0.9722222 | 0.9571429 | 0.9142857 | 1.0000000 | 1584465255 | 0.9981459 | 0.9976831 | 0.9986087 | 0.9939555 | 1.0000000 | 0.9879708 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9714286 | 1.0000000 | 0.9428571 | 1584465759 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9444444 | 0.9444444 | 0.9444444 | 0.9857143 | 0.9714286 | 1.0000000 | 1584466567 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9305556 | 0.9444444 | 0.9166667 | 1.0000000 | 1.0000000 | 1.0000000 | 1584467303 | 0.9774786 | 0.9755956 | 0.9793616 | 0.9755893 | 1.0000000 | 0.9514204 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584467689 | 0.9909485 | 0.9897382 | 0.9921589 | 0.9901892 | 1.0000000 | 0.9804756 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584468088 | 0.9999952 | 0.9999895 | 1.0000000 | 0.9998140 | 1.0000000 | 0.9996299 | 0.8888889 | 0.8333333 | 0.9444444 | 0.9285714 | 0.8857143 | 0.9714286 |
| 21 | SU_MDLSMOTE | Yes | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.708.3p + hsa.miR.375.3p + hsa.miR.30c.2.3p + hsa.miR.217.5p + hsa.miR.194.5p + hsa.miR.200a.3p + hsa.miR.132.3p + hsa.miR.199a.5p | 1584463142 | 0.9989010 | 0.9985914 | 0.9992106 | 0.9940949 | 1.0000000 | 0.9882484 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463622 | 0.9958725 | 0.9950065 | 0.9967385 | 0.9939090 | 1.0000000 | 0.9878782 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464140 | 0.9984466 | 0.9979116 | 0.9989817 | 0.9966523 | 1.0000000 | 0.9933377 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464730 | 0.9994697 | 0.9991769 | 0.9997624 | 0.9982796 | 1.0000000 | 0.9965763 | 0.9444444 | 0.9444444 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465273 | 0.9984868 | 0.9980649 | 0.9989088 | 0.9947459 | 1.0000000 | 0.9895438 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465791 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9305556 | 0.9166667 | 0.9444444 | 0.9428571 | 0.8857143 | 1.0000000 | 1584466634 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9999535 | 1.0000000 | 0.9999075 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9571429 | 0.9428571 | 0.9714286 | 1584467320 | 0.9968461 | 0.9961296 | 0.9975626 | 0.9965128 | 1.0000000 | 0.9930601 | 0.9583333 | 0.9722222 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584467705 | 0.9909485 | 0.9897382 | 0.9921589 | 0.9901892 | 1.0000000 | 0.9804756 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584468107 | 0.9999760 | 0.9999595 | 0.9999924 | 0.9993955 | 1.0000000 | 0.9987971 | 0.8750000 | 0.8055556 | 0.9444444 | 0.9428571 | 0.9142857 | 0.9714286 |
| 22 | CorrSF_MDLSMOTE | Yes | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.708.3p + hsa.miR.30a.3p + hsa.miR.136.5p + hsa.miR.132.3p | 1584463161 | 0.9969292 | 0.9962894 | 0.9975690 | 0.9845632 | 0.9913084 | 0.9778847 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463646 | 0.9921348 | 0.9909225 | 0.9933470 | 0.9887014 | 0.9913084 | 0.9861201 | 0.9583333 | 0.9722222 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584464166 | 0.9964133 | 0.9957401 | 0.9970865 | 0.9889338 | 0.9913084 | 0.9865828 | 0.9583333 | 0.9722222 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584464763 | 0.9993308 | 0.9989900 | 0.9996717 | 0.9972102 | 1.0000000 | 0.9944480 | 0.9305556 | 0.9166667 | 0.9444444 | 0.9285714 | 0.8571429 | 1.0000000 | 1584465293 | 0.9964932 | 0.9958250 | 0.9971615 | 0.9863300 | 0.9913084 | 0.9814009 | 0.9444444 | 0.9722222 | 0.9166667 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465814 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9305556 | 0.8888889 | 0.9722222 | 0.9714286 | 0.9428571 | 1.0000000 | 1584466700 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9996280 | 1.0000000 | 0.9992597 | 0.9305556 | 0.9166667 | 0.9444444 | 0.9857143 | 0.9714286 | 1.0000000 | 1584467336 | 0.9729217 | 0.9708267 | 0.9750166 | 0.9712652 | 0.9913084 | 0.9514204 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584467722 | 0.9866442 | 0.9851528 | 0.9881355 | 0.9858651 | 0.9913084 | 0.9804756 | 0.9305556 | 0.9444444 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584468126 | 0.9999149 | 0.9998332 | 0.9999967 | 0.9992096 | 1.0000000 | 0.9984269 | 0.9305556 | 0.9166667 | 0.9444444 | 0.9428571 | 0.9428571 | 0.9428571 |
| 23 | AUC_MDL_sig | No | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.139.5p + hsa.miR.127.5p | 1584463178 | 0.9931872 | 0.9810931 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463659 | 0.9902612 | 0.9763623 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464178 | 0.9896934 | 0.9746777 | 1.0000000 | 0.9813084 | 0.9813084 | 0.9813084 | 0.9722222 | 1.0000000 | 0.9444444 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464781 | 0.9910036 | 0.9740380 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9583333 | 0.9722222 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465307 | 0.9894314 | 0.9729248 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465826 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584466713 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584467350 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467736 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584468143 | 0.9941480 | 0.9887442 | 0.9995518 | 0.9579439 | 0.9345794 | 0.9813084 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9571429 | 0.9714286 | 0.9428571 |
| 24 | SU_MDL_sig | No | Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30e.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.30a.3p + hsa.miR.10b.5p + hsa.miR.126.5p + hsa.miR.199b.5p + hsa.miR.30c.2.3p + hsa.miR.126.3p | 1584463185 | 0.9965936 | 0.9915862 | 1.0000000 | 0.9766355 | 0.9813084 | 0.9719626 | 0.9305556 | 0.9444444 | 0.9166667 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463666 | 0.9928378 | 0.9815851 | 1.0000000 | 0.9813084 | 0.9906542 | 0.9719626 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584464186 | 0.9925758 | 0.9812489 | 1.0000000 | 0.9392523 | 1.0000000 | 0.8785047 | 0.8888889 | 1.0000000 | 0.7777778 | 0.9142857 | 1.0000000 | 0.8285714 | 1584464789 | 0.9975544 | 0.9928587 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9583333 | 0.9722222 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465315 | 0.9937986 | 0.9837316 | 1.0000000 | 0.9813084 | 0.9906542 | 0.9719626 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 1.0000000 | 0.9428571 | 1584465834 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584466721 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9722222 | 1.0000000 | 0.9444444 | 0.9571429 | 0.9714286 | 0.9428571 | 1584467358 | 0.9889073 | 0.9755302 | 1.0000000 | 0.9672897 | 0.9439252 | 0.9906542 | 0.9305556 | 0.9444444 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467744 | 0.9889073 | 0.9755302 | 1.0000000 | 0.9672897 | 0.9439252 | 0.9906542 | 0.9305556 | 0.9444444 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584468151 | 0.9825749 | 0.9652791 | 0.9998707 | 0.9579439 | 0.9345794 | 0.9813084 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9571429 | 0.9714286 | 0.9428571 |
| 25 | CorrSF_MDL_sig | No | Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30e.3p + hsa.miR.194.5p + hsa.miR.10b.5p + hsa.miR.199b.5p + hsa.miR.30c.2.3p + hsa.miR.338.3p + hsa.miR.217.5p + hsa.miR.378a.5p + hsa.miR.195.5p | 1584463193 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463674 | 0.9969430 | 0.9916498 | 1.0000000 | 0.9766355 | 0.9719626 | 0.9813084 | 0.9583333 | 0.9722222 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584464194 | 0.9962442 | 0.9898840 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464797 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9583333 | 0.9722222 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465323 | 0.9976417 | 0.9931160 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465842 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584466729 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9722222 | 1.0000000 | 0.9444444 | 0.9571429 | 0.9714286 | 0.9428571 | 1584467366 | 0.9889073 | 0.9755302 | 1.0000000 | 0.9672897 | 0.9439252 | 0.9906542 | 0.9166667 | 0.9166667 | 0.9166667 | 0.9571429 | 0.9714286 | 0.9428571 | 1584467752 | 0.9889073 | 0.9755302 | 1.0000000 | 0.9672897 | 0.9439252 | 0.9906542 | 0.9166667 | 0.9166667 | 0.9166667 | 0.9571429 | 0.9714286 | 0.9428571 | 1584468158 | 0.9942790 | 0.9890146 | 0.9995433 | 0.9579439 | 0.9345794 | 0.9813084 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9571429 | 0.9714286 | 0.9428571 |
| 26 | AUC_MDLSMOTE_sig | Yes | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.132.3p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.127.5p + hsa.miR.139.5p | 1584463202 | 0.9981693 | 0.9976671 | 0.9986715 | 0.9943739 | 1.0000000 | 0.9888036 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463695 | 0.9948446 | 0.9938796 | 0.9958095 | 0.9927930 | 1.0000000 | 0.9856574 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464216 | 0.9997759 | 0.9996748 | 0.9998771 | 0.9961408 | 1.0000000 | 0.9923198 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464818 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9999535 | 1.0000000 | 0.9999075 | 0.9305556 | 0.8888889 | 0.9722222 | 0.9571429 | 0.9142857 | 1.0000000 | 1584465334 | 0.9975346 | 0.9968667 | 0.9982025 | 0.9940484 | 1.0000000 | 0.9881558 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465869 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9583333 | 0.9444444 | 0.9722222 | 1.0000000 | 1.0000000 | 1.0000000 | 1584466792 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9444444 | 0.9722222 | 0.9166667 | 1.0000000 | 1.0000000 | 1.0000000 | 1584467375 | 0.9774786 | 0.9755956 | 0.9793616 | 0.9755893 | 1.0000000 | 0.9514204 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584467761 | 0.9909485 | 0.9897382 | 0.9921589 | 0.9901892 | 1.0000000 | 0.9804756 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584468167 | 0.9999504 | 0.9998922 | 1.0000000 | 0.9993490 | 1.0000000 | 0.9987045 | 0.9027778 | 0.8611111 | 0.9444444 | 0.9428571 | 0.9142857 | 0.9714286 |
| 27 | SU_MDLSMOTE_sig | Yes | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.375.3p + hsa.miR.708.3p + hsa.miR.136.5p + hsa.miR.30c.2.3p + hsa.miR.194.5p + hsa.miR.200a.3p + hsa.miR.217.5p + hsa.miR.132.3p | 1584463221 | 0.9991771 | 0.9989537 | 0.9994004 | 0.9942344 | 1.0000000 | 0.9885260 | 0.9583333 | 0.9722222 | 0.9444444 | 0.9857143 | 1.0000000 | 0.9714286 | 1584463720 | 0.9947933 | 0.9938227 | 0.9957639 | 0.9901892 | 1.0000000 | 0.9804756 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584464241 | 0.9999494 | 0.9999143 | 0.9999845 | 0.9993490 | 1.0000000 | 0.9987045 | 0.9444444 | 0.9722222 | 0.9166667 | 1.0000000 | 1.0000000 | 1.0000000 | 1584464850 | 0.9998279 | 0.9996898 | 0.9999661 | 0.9993026 | 1.0000000 | 0.9986120 | 0.9583333 | 0.9444444 | 0.9722222 | 0.9714286 | 0.9428571 | 1.0000000 | 1584465353 | 0.9989748 | 0.9986937 | 0.9992560 | 0.9946994 | 1.0000000 | 0.9894513 | 0.9583333 | 0.9722222 | 0.9444444 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465899 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9583333 | 0.9722222 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584466860 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9305556 | 0.9444444 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584467392 | 0.9968461 | 0.9961296 | 0.9975626 | 0.9965128 | 1.0000000 | 0.9930601 | 0.9583333 | 0.9722222 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584467778 | 0.9909485 | 0.9897382 | 0.9921589 | 0.9901892 | 1.0000000 | 0.9804756 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584468186 | 0.9999760 | 0.9999595 | 0.9999924 | 0.9993955 | 1.0000000 | 0.9987971 | 0.8750000 | 0.8055556 | 0.9444444 | 0.9428571 | 0.9142857 | 0.9714286 |
| 28 | CorrSF_MDLSMOTE_sig | Yes | Class ~ hsa.miR.21.5p + hsa.miR.132.3p + hsa.miR.30a.3p | 1584463240 | 0.9914422 | 0.9902532 | 0.9926312 | 0.9824708 | 0.9913084 | 0.9737207 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463745 | 0.9917561 | 0.9906045 | 0.9929077 | 0.9810759 | 0.9913084 | 0.9709448 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9714286 | 1.0000000 | 0.9428571 | 1584464266 | 0.9918466 | 0.9907094 | 0.9929837 | 0.9732180 | 0.9652336 | 0.9811233 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584464888 | 0.9968592 | 0.9961564 | 0.9975620 | 0.9924211 | 1.0000000 | 0.9849172 | 0.9305556 | 0.9166667 | 0.9444444 | 0.9571429 | 0.9428571 | 0.9714286 | 1584465373 | 0.9906387 | 0.9893655 | 0.9919119 | 0.9823778 | 0.9913084 | 0.9735357 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465919 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9305556 | 0.8888889 | 0.9722222 | 0.9142857 | 0.8571429 | 0.9714286 | 1584466923 | 0.9998515 | 0.9997596 | 0.9999434 | 0.9989771 | 1.0000000 | 0.9979643 | 0.9166667 | 0.8888889 | 0.9444444 | 0.9428571 | 0.9142857 | 0.9714286 | 1584467408 | 0.9729217 | 0.9708267 | 0.9750166 | 0.9712652 | 0.9913084 | 0.9514204 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584467794 | 0.9729217 | 0.9708267 | 0.9750166 | 0.9712652 | 0.9913084 | 0.9514204 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584468205 | 0.9996431 | 0.9995127 | 0.9997735 | 0.9974892 | 1.0000000 | 0.9950032 | 0.9444444 | 0.9166667 | 0.9722222 | 0.9428571 | 0.9428571 | 0.9428571 |
| 29 | RandomForestRFE | No | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.375.3p + hsa.miR.132.3p + hsa.miR.30c.2.3p + hsa.miR.217.5p + hsa.miR.194.5p + hsa.miR.192.5p + hsa.miR.126.5p | 1584463255 | 0.9963316 | 0.9902920 | 1.0000000 | 0.9859813 | 1.0000000 | 0.9719626 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463757 | 0.9917897 | 0.9780033 | 1.0000000 | 0.9813084 | 0.9813084 | 0.9813084 | 0.9722222 | 1.0000000 | 0.9444444 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464279 | 0.9910909 | 0.9766273 | 1.0000000 | 0.9719626 | 0.9626168 | 0.9813084 | 0.9722222 | 1.0000000 | 0.9444444 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464905 | 0.9955455 | 0.9867696 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9722222 | 1.0000000 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465387 | 0.9923137 | 0.9797836 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465931 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584466934 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584467421 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467807 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584468221 | 0.9941480 | 0.9887442 | 0.9995518 | 0.9579439 | 0.9345794 | 0.9813084 | 0.9166667 | 0.9444444 | 0.8888889 | 0.9571429 | 0.9714286 | 0.9428571 |
| 30 | cfsSMOTE | Yes | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.132.3p + hsa.miR.136.5p + hsa.miR.30e.3p | 1584463264 | 0.9965618 | 0.9958880 | 0.9972356 | 0.9843307 | 0.9913084 | 0.9774220 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463777 | 0.9956546 | 0.9948894 | 0.9964198 | 0.9888408 | 0.9913084 | 0.9863977 | 0.9444444 | 0.9444444 | 0.9444444 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464299 | 0.9913708 | 0.9900964 | 0.9926452 | 0.9886084 | 0.9913084 | 0.9859350 | 0.9722222 | 1.0000000 | 0.9444444 | 0.9714286 | 1.0000000 | 0.9428571 | 1584464929 | 0.9982670 | 0.9977489 | 0.9987851 | 0.9942344 | 1.0000000 | 0.9885260 | 0.9305556 | 0.9166667 | 0.9444444 | 0.9571429 | 0.9428571 | 0.9714286 | 1584465400 | 0.9957081 | 0.9949285 | 0.9964877 | 0.9865625 | 0.9913084 | 0.9818636 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9571429 | 1.0000000 | 0.9142857 | 1584465950 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9166667 | 0.8888889 | 0.9444444 | 0.9428571 | 0.8857143 | 1.0000000 | 1584466995 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9997675 | 1.0000000 | 0.9995373 | 0.9305556 | 0.9166667 | 0.9444444 | 0.9571429 | 0.9428571 | 0.9714286 | 1584467430 | 0.9729217 | 0.9708267 | 0.9750166 | 0.9712652 | 0.9913084 | 0.9514204 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584467816 | 0.9866442 | 0.9851528 | 0.9881355 | 0.9858651 | 0.9913084 | 0.9804756 | 0.9305556 | 0.9444444 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584468230 | 0.9997259 | 0.9995436 | 0.9999082 | 0.9989306 | 1.0000000 | 0.9978717 | 0.9444444 | 0.9444444 | 0.9444444 | 0.9000000 | 0.8571429 | 0.9428571 |
| 31 | cfsSMOTE_sig | Yes | Class ~ hsa.miR.21.5p + hsa.miR.30e.3p + hsa.miR.30a.3p + hsa.miR.132.3p + hsa.miR.136.5p | 1584463281 | 0.9965618 | 0.9958880 | 0.9972356 | 0.9843307 | 0.9913084 | 0.9774220 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 1.0000000 | 0.9428571 | 1584463801 | 0.9922008 | 0.9909959 | 0.9934058 | 0.9764728 | 0.9639252 | 0.9888961 | 0.9444444 | 0.9444444 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584464322 | 0.9909486 | 0.9896368 | 0.9922603 | 0.9870740 | 0.9913084 | 0.9828815 | 0.9722222 | 1.0000000 | 0.9444444 | 0.9714286 | 1.0000000 | 0.9428571 | 1584464961 | 0.9982681 | 0.9977503 | 0.9987860 | 0.9941879 | 1.0000000 | 0.9884334 | 0.9305556 | 0.9166667 | 0.9444444 | 0.9571429 | 0.9428571 | 0.9714286 | 1584465419 | 0.9957081 | 0.9949285 | 0.9964877 | 0.9866090 | 0.9913084 | 0.9819561 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9571429 | 1.0000000 | 0.9142857 | 1584465973 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9166667 | 0.8888889 | 0.9444444 | 0.9428571 | 0.8857143 | 1.0000000 | 1584467060 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9997675 | 1.0000000 | 0.9995373 | 0.9305556 | 0.9166667 | 0.9444444 | 0.9571429 | 0.9428571 | 0.9714286 | 1584467445 | 0.9729217 | 0.9708267 | 0.9750166 | 0.9712652 | 0.9913084 | 0.9514204 | 0.9305556 | 0.9722222 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584467831 | 0.9866442 | 0.9851528 | 0.9881355 | 0.9858651 | 0.9913084 | 0.9804756 | 0.9305556 | 0.9444444 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584468247 | 0.9997259 | 0.9995436 | 0.9999082 | 0.9989306 | 1.0000000 | 0.9978717 | 0.9444444 | 0.9444444 | 0.9444444 | 0.9000000 | 0.8571429 | 0.9428571 |
| 32 | Mystepwise_glm_binomial | No | Class ~ hsa.miR.21.5p + hsa.miR.130a.3p + hsa.miR.30e.3p + hsa.miR.375.3p + hsa.miR.378a.3p + hsa.miR.146b.5p | 1584463298 | 0.9995633 | 0.9987652 | 1.0000000 | 0.9906542 | 0.9906542 | 0.9906542 | 0.9583333 | 0.9722222 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584463814 | 0.9971177 | 0.9924022 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464335 | 0.9976417 | 0.9938877 | 1.0000000 | 0.9906542 | 1.0000000 | 0.9813084 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584464978 | 0.9994759 | 0.9983702 | 1.0000000 | 0.9906542 | 0.9906542 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465434 | 0.9972050 | 0.9925361 | 1.0000000 | 0.9859813 | 0.9906542 | 0.9813084 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465985 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584467072 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584467458 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584467845 | 0.9939296 | 0.9848574 | 1.0000000 | 0.9719626 | 0.9532710 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9714286 | 0.9714286 | 0.9714286 | 1584468263 | 0.9945410 | 0.9895576 | 0.9995244 | 0.9579439 | 0.9345794 | 0.9813084 | 0.9166667 | 0.9166667 | 0.9166667 | 0.9428571 | 0.9714286 | 0.9142857 |
| 33 | Mystepwise_sig_glm_binomial | No | Class ~ hsa.miR.21.5p + hsa.miR.26b.5p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.378a.3p + hsa.miR.338.3p | 1584463306 | 0.9999127 | 0.9996706 | 1.0000000 | 0.9906542 | 0.9906542 | 0.9906542 | 0.9583333 | 1.0000000 | 0.9166667 | 1.0000000 | 1.0000000 | 1.0000000 | 1584463821 | 0.9998253 | 0.9994066 | 1.0000000 | 0.9906542 | 0.9813084 | 1.0000000 | 0.9444444 | 0.9722222 | 0.9166667 | 1.0000000 | 1.0000000 | 1.0000000 | 1584464343 | 0.9997380 | 0.9991987 | 1.0000000 | 0.9906542 | 0.9906542 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 1.0000000 | 1.0000000 | 1.0000000 | 1584464987 | 0.9998253 | 0.9994066 | 1.0000000 | 0.9953271 | 1.0000000 | 0.9906542 | 0.9722222 | 1.0000000 | 0.9444444 | 1.0000000 | 1.0000000 | 1.0000000 | 1584465442 | 0.9997380 | 0.9991987 | 1.0000000 | 0.9906542 | 0.9906542 | 0.9906542 | 0.9444444 | 0.9722222 | 0.9166667 | 0.9857143 | 1.0000000 | 0.9714286 | 1584465993 | 1 | 1 | 1 | 1 | 1 | 1 | 0.9722222 | 1.0000000 | 0.9444444 | 0.9857143 | 1.0000000 | 0.9714286 | 1584467080 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 1.0000000 | 0.9722222 | 1.0000000 | 0.9444444 | 0.9857143 | 1.0000000 | 0.9714286 | 1584467466 | 0.9816578 | 0.9643889 | 0.9989267 | 0.9672897 | 0.9719626 | 0.9626168 | 0.9861111 | 1.0000000 | 0.9722222 | 0.9857143 | 1.0000000 | 0.9714286 | 1584467853 | 0.9654118 | 0.9421733 | 0.9886504 | 0.9626168 | 1.0000000 | 0.9252336 | 0.9444444 | 1.0000000 | 0.8888889 | 0.9714286 | 1.0000000 | 0.9428571 | 1584468270 | 0.9888200 | 0.9767842 | 1.0000000 | 0.9532710 | 0.9532710 | 0.9532710 | 0.9583333 | 1.0000000 | 0.9166667 | 0.9428571 | 1.0000000 | 0.8857143 |
Description of columns:
method - accronym for methodSMOTE - if balancing using SMOTE or ROSE was used in training dataset.miRy - formula used (miRNAs selected).*_modelname - the name of .RDS file placed in models/ directory, containing the caret final model that can be used for prediction of new cases. This allows reproducibility of the results. For example glm_ prefix is set according to the method, glm = logistic regression.*_train_ROCAUC - area under the ROC curve (AUC ROC) on training dataset, indicating general potential of the model.*_train_ROCAUC_lower95CI - lower boundery of 95% confidence interval for AUC ROC.*_train_ROCAUC_upper95CI - upper boundery of 95% confidence interval for AUC ROC.*_train_Accuracy - accuracy on training set.*_train_Sensitivity - sensitivity on training set.*_train_Specificity - sensitivity on training set.By this logic every parameter is also calculated from testing (_test_) and validation (_valid_) set. If the method generated a probability, a default cutoff is used for all of the predictions.
Let’s see the general performance (accuracy) of methods in the benchmark:
metody = ks.get_benchmark_methods("benchmark.csv") # gets the methods used in benchmark
par(mfrow = c(2, 2))
for (i in 1:length(metody)) {
temp = ks.get_benchmark("benchmark.csv") # loads benchmark
temp2 = dplyr::select(temp, starts_with(paste0(metody[i], "_")))
boxplot(temp[, paste0(metody[i], "_train_Accuracy")], temp[, paste0(metody[i], "_test_Accuracy")], temp[, paste0(metody[i], "_valid_Accuracy")], main = metody[i],
names = c("Training", "Testing", "Validation"), ylab = "Accuracy", ylim = c(0.5, 1))
tempids = c(match(paste0(metody[i], "_train_Accuracy"), colnames(temp)), match(paste0(metody[i], "_test_Accuracy"), colnames(temp)), match(paste0(metody[i],
"_valid_Accuracy"), colnames(temp)))
}In this package, the best signature can be selected using 3 methods:
1. The signture which achived the best accuracy in training, testing and validation: (metaindex = mean of all 3 accuracy metrics)
acc1 = ks.best_signiture_proposals(benchmark_csv = "benchmark.csv", without_train = F) # generates the benchmark sorted by metaindex
best_signatures = acc1[1:3, ] # get top 3 methods
knitr::kable(best_signatures[, 31:33])| metaindex | method | miRy |
|---|---|---|
| 0.9775265 | topFCSMOTE | Class ~ hsa.miR.217.5p + hsa.miR.375.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.199b.5p + hsa.miR.144.5p + hsa.miR.30c.2.3p + hsa.miR.200a.3p + hsa.miR.708.3p |
| 0.9765162 | Mystepwise_sig_glm_binomial | Class ~ hsa.miR.21.5p + hsa.miR.26b.5p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.378a.3p + hsa.miR.338.3p |
| 0.9741000 | SU_MDLSMOTE_sig | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.375.3p + hsa.miR.708.3p + hsa.miR.136.5p + hsa.miR.30c.2.3p + hsa.miR.194.5p + hsa.miR.200a.3p + hsa.miR.217.5p + hsa.miR.132.3p |
2. The signture which achived the best accuracy in testing and validation: (metaindex = mean of 2 accuracy metrics)
acc1 = ks.best_signiture_proposals(benchmark_csv = "benchmark.csv", without_train = T) # generates the benchmark sorted by metaindex
best_signatures = acc1[1:3, ] # get top 3 methods
knitr::kable(best_signatures[, 21:23])| metaindex | method | miRy |
|---|---|---|
| 0.9727183 | Mystepwise_sig_glm_binomial | Class ~ hsa.miR.21.5p + hsa.miR.26b.5p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.378a.3p + hsa.miR.338.3p |
| 0.9685913 | topFCSMOTE | Class ~ hsa.miR.217.5p + hsa.miR.375.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.199b.5p + hsa.miR.144.5p + hsa.miR.30c.2.3p + hsa.miR.200a.3p + hsa.miR.708.3p |
| 0.9671429 | AUC_MDL | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.139.5p + hsa.miR.127.5p |
3. The signture which achived the best sensitivity and specificity in validation: (metaindex = mean of sensivitiy and specificity in validation dataset)
acc = ks.best_signiture_proposals_meta11(benchmark_csv = "benchmark.csv") # generates the benchmark sorted by metaindex
best_signatures = acc[1:3, ] # get top 3 methods
knitr::kable(best_signatures[, c(2:4, 135)])| method | SMOTE | miRy | metaindex |
|---|---|---|---|
| topFCSMOTE | Yes | Class ~ hsa.miR.217.5p + hsa.miR.375.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.199b.5p + hsa.miR.144.5p + hsa.miR.30c.2.3p + hsa.miR.200a.3p + hsa.miR.708.3p | 0.9771429 |
| Mystepwise_sig_glm_binomial | No | Class ~ hsa.miR.21.5p + hsa.miR.26b.5p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.378a.3p + hsa.miR.338.3p | 0.9714286 |
| AUC_MDL | No | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.139.5p + hsa.miR.127.5p | 0.9685714 |
Let’s assess the over/underfitting of selected methods for top 2 signatures. The plots show the change of accuracy between datasets across top 5 methods of feature selection.
for (i in 1:length(metody)) {
suppressMessages(library(PairedData))
suppressMessages(library(profileR))
pd = paired(as.numeric(acc[1:5, paste0(metody[i], "_train_Accuracy")]), as.numeric(acc[1:5, paste0(metody[i], "_test_Accuracy")]))
colnames(pd) = c("Train Acc", "Test Acc")
plot2 = profileplot(pd, person.id = acc$method[1:5], standardize = F)
pd = paired(as.numeric(acc[1:5, paste0(metody[i], "_train_Accuracy")]), as.numeric(acc[1:5, paste0(metody[i], "_valid_Accuracy")]))
colnames(pd) = c("Train Acc", "Valid Acc")
plot3 = profileplot(pd, person.id = acc$method[1:5], standardize = F)
pd = paired(as.numeric(acc[1:5, paste0(metody[i], "_test_Accuracy")]), as.numeric(acc[1:5, paste0(metody[i], "_valid_Accuracy")]))
colnames(pd) = c("Test Acc", "Valid Acc")
plot4 = profileplot(pd, person.id = acc$method[1:5], standardize = F)
require(gridExtra)
grid.arrange(arrangeGrob(plot2, plot3, ncol = 2, nrow = 1, top = metody[i]))
grid.arrange(arrangeGrob(plot4, ncol = 1, nrow = 1, top = metody[i]))
}The relationship betweend accuracy on testin and training set can be further visualized as follows:
acc2 = acc[1:6, ] # get top 6 methods
accmelt = melt(acc2, id.vars = "method") %>% filter(variable != "metaindex") %>% filter(variable != "miRy")
accmelt = cbind(accmelt, strsplit2(accmelt$variable, "_"))
acctest = accmelt$value[accmelt$`2` == "test"]
accvalid = accmelt$value[accmelt$`2` == "valid"]
accmeth = accmelt$method[accmelt$`2` == "test"]
unique(accmeth)
#> [1] "topFCSMOTE" "Mystepwise_sig_glm_binomial" "AUC_MDL" "SU_MDLSMOTE_sig" "cfs_sig"
#> [6] "AUC_MDL_sig"
plot5 = ggplot(, aes(x = as.numeric(acctest), y = as.numeric(accvalid), shape = accmeth)) + geom_point() + scale_x_continuous(name = "Accuracy on test set",
limits = c(0.5, 1)) + scale_y_continuous(name = "Accuracy on validation set", limits = c(0.5, 1)) + theme_bw()
grid.arrange(arrangeGrob(plot5, ncol = 1, nrow = 1))Suppose we chose to select the best signitures based on the best sensitivity and specificity in validation. Let’s see 3 best signutures:
| method | SMOTE | miRy |
|---|---|---|
| topFCSMOTE | Yes | Class ~ hsa.miR.217.5p + hsa.miR.375.3p + hsa.miR.192.5p + hsa.miR.194.5p + hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.199b.5p + hsa.miR.144.5p + hsa.miR.30c.2.3p + hsa.miR.200a.3p + hsa.miR.708.3p |
| Mystepwise_sig_glm_binomial | No | Class ~ hsa.miR.21.5p + hsa.miR.26b.5p + hsa.miR.30c.2.3p + hsa.miR.375.3p + hsa.miR.378a.3p + hsa.miR.338.3p |
| AUC_MDL | No | Class ~ hsa.miR.21.5p + hsa.miR.30a.3p + hsa.miR.30e.3p + hsa.miR.30c.2.3p + hsa.miR.132.3p + hsa.miR.375.3p + hsa.miR.217.5p + hsa.miR.136.5p + hsa.miR.199a.5p + hsa.miR.139.5p + hsa.miR.127.5p |
To get the miRNAs from formula you can use ks.get_miRNAs_from_benchmark.
selected_miRNAs = ks.get_miRNAs_from_benchmark(benchmark_csv = "benchmark.csv", best_signatures$method[1]) # for the best performing signiture
selected_miRNAs
#> [1] "hsa.miR.217.5p" "hsa.miR.375.3p" "hsa.miR.192.5p" "hsa.miR.194.5p" "hsa.miR.21.5p" "hsa.miR.30a.3p" "hsa.miR.199b.5p" "hsa.miR.144.5p" "hsa.miR.30c.2.3p"
#> [10] "hsa.miR.200a.3p" "hsa.miR.708.3p"Let’s check the differential expression metric of selected miRNAs:
| miR | log2FC | p-value BH | |
|---|---|---|---|
| 7 | hsa.miR.217.5p | 4.996853 | 0 |
| 6 | hsa.miR.375.3p | 4.788105 | 0 |
| 23 | hsa.miR.192.5p | 2.963697 | 0 |
| 18 | hsa.miR.194.5p | 2.943915 | 0 |
| 1 | hsa.miR.21.5p | 2.602216 | 0 |
| 3 | hsa.miR.30a.3p | -2.441965 | 0 |
| 11 | hsa.miR.199b.5p | 2.381457 | 0 |
| 39 | hsa.miR.144.5p | -2.112787 | 0 |
| 4 | hsa.miR.30c.2.3p | -2.082431 | 0 |
| 25 | hsa.miR.200a.3p | 2.194942 | 0 |
| 30 | hsa.miR.708.3p | 2.162216 | 0 |
Let’s visualize the performance of those methods using barplots:
for (i in 1:3) {
cat(paste0("\n\n## ", acc$method[i], "\n\n"))
par(mfrow = c(1, 2))
acc = ks.best_signiture_proposals_meta11("benchmark.csv")
metody = ks.get_benchmark_methods("benchmark.csv")
ktory_set = match(acc$method[i], ks.get_benchmark("benchmark.csv")$method)
# do_ktorej_kolumny = which(colnames(acc) == 'metaindex') barplot(as.numeric(acc[i,1:do_ktorej_kolumny]))
for (ii in 1:length(metody)) {
temp = ks.get_benchmark("benchmark.csv") %>% dplyr::select(starts_with(paste0(metody[ii], "_t")), starts_with(paste0(metody[ii], "_v")))
ROCtext = paste0("Training AUC ROC: ", round(temp[ktory_set, 1], 2), " (95%CI: ", round(temp[ktory_set, 2], 2), "-", round(temp[ktory_set, 3],
2), ")")
temp = temp[, -c(1:3)]
temp2 = as.numeric(temp[ktory_set, ])
temp3 = matrix(temp2, nrow = 3, byrow = T)
colnames(temp3) = c("Accuracy", "Sensitivity", "Specificity")
rownames(temp3) = c("Training", "Testing", "Validation")
temp3 = t(temp3)
plot1 = barplot(temp3, beside = T, ylim = c(0, 1), xlab = paste0(ROCtext, "\nBlack - accuracy, blue - sensitivity, green - specificity"), width = 0.85,
col = c("black", "blue", "green"), legend = F, args.legend = list(x = "topright", bty = "n", inset = c(0, -0.25)), cex.lab = 0.7, main = paste0(acc$method[i],
" - ", metody[ii]), font.lab = 2)
## Add text at top of bars
text(x = plot1, y = as.numeric(temp3), label = paste0(round(as.numeric(temp[ktory_set, ]) * 100, 1), "%"), pos = 3, cex = 0.6, col = "red")
}
par(mfrow = c(1, 1))
}
#>
#>
#> ## topFCSMOTE#>
#>
#> ## Mystepwise_sig_glm_binomial
#>
#>
#> ## AUC_MDL
Finally, we can assess the overlap of top 3 feature selection methods:
Which 3 miRNAs are common for all 3 signatures?
attr(overlap, "intersections")$`topFCSMOTE:Mystepwise_sig_glm_binomial:AUC_MDL`
#> [1] "hsa.miR.375.3p" "hsa.miR.21.5p" "hsa.miR.30c.2.3p"Let’s draw vulcano plot and mark the miRNAs selected in best signature:
Let’s draw heatmap for selected miRNAs in whole dataset (training, testing and validation set).
Based on everything we have done so far, we suggest using the following signiture in further validation of biomarker study.
session_info()
#> ─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> setting value
#> version R version 3.6.2 (2019-12-12)
#> os Ubuntu 18.04.4 LTS
#> system x86_64, linux-gnu
#> ui RStudio
#> language (EN)
#> collate en_US.UTF-8
#> ctype en_US.UTF-8
#> tz Europe/Warsaw
#> date 2020-03-18
#>
#> ─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> package * version date lib source
#> abind 1.4-5 2016-07-21 [1] CRAN (R 3.6.0)
#> arules 1.6-4 2019-08-29 [1] CRAN (R 3.6.1)
#> assertthat 0.2.1 2019-03-21 [1] CRAN (R 3.6.0)
#> backports 1.1.5 2019-10-02 [1] CRAN (R 3.6.1)
#> Biocomb * 0.4 2018-05-18 [1] CRAN (R 3.6.2)
#> bitops 1.0-6 2013-08-17 [1] CRAN (R 3.6.0)
#> boot 1.3-24 2019-12-20 [4] CRAN (R 3.6.2)
#> bounceR 0.1 2019-07-29 [1] Github (STATWORX/bounceR@0f714f3)
#> broom 0.5.5 2020-02-29 [1] CRAN (R 3.6.2)
#> callr 3.4.2 2020-02-12 [1] CRAN (R 3.6.2)
#> caret * 6.0-85 2020-01-07 [1] CRAN (R 3.6.2)
#> caTools 1.18.0 2020-01-17 [1] CRAN (R 3.6.1)
#> cellranger 1.1.0 2016-07-27 [1] CRAN (R 3.6.0)
#> class 7.3-15 2019-01-01 [4] CRAN (R 3.6.0)
#> classInt * 0.4-2 2019-10-17 [1] CRAN (R 3.6.2)
#> cli 2.0.2 2020-02-28 [1] CRAN (R 3.6.2)
#> cluster 2.1.0 2019-06-19 [1] CRAN (R 3.6.1)
#> codetools 0.2-16 2018-12-24 [4] CRAN (R 3.6.0)
#> colorspace 1.4-1 2019-03-18 [1] CRAN (R 3.6.0)
#> crayon 1.3.4 2017-09-16 [1] CRAN (R 3.6.0)
#> crosstalk 1.1.0.1 2020-03-13 [1] CRAN (R 3.6.2)
#> cubature 2.0.4 2019-12-04 [1] CRAN (R 3.6.1)
#> curl 4.3 2019-12-02 [1] CRAN (R 3.6.1)
#> data.table * 1.12.8 2019-12-09 [1] CRAN (R 3.6.2)
#> DBI 1.1.0 2019-12-15 [1] CRAN (R 3.6.1)
#> dbplyr 1.4.2 2019-06-17 [1] CRAN (R 3.6.1)
#> desc 1.2.0 2018-05-01 [1] CRAN (R 3.6.0)
#> devtools * 2.2.2 2020-02-17 [1] CRAN (R 3.6.2)
#> digest 0.6.25 2020-02-23 [1] CRAN (R 3.6.2)
#> DMwR * 0.4.1 2013-08-08 [1] CRAN (R 3.6.2)
#> doParallel * 1.0.15 2019-08-02 [1] CRAN (R 3.6.2)
#> dplyr * 0.8.5 2020-03-07 [1] CRAN (R 3.6.2)
#> e1071 1.7-3 2019-11-26 [1] CRAN (R 3.6.1)
#> edgeR * 3.26.8 2019-09-01 [1] Bioconductor
#> ellipsis 0.3.0 2019-09-20 [1] CRAN (R 3.6.1)
#> entropy 1.2.1 2014-11-14 [1] CRAN (R 3.6.1)
#> epiDisplay * 3.5.0.1 2018-05-10 [1] CRAN (R 3.6.2)
#> evaluate 0.14 2019-05-28 [1] CRAN (R 3.6.0)
#> fansi 0.4.1 2020-01-08 [1] CRAN (R 3.6.1)
#> farver 2.0.3 2020-01-16 [1] CRAN (R 3.6.1)
#> fastmap 1.0.1 2019-10-08 [1] CRAN (R 3.6.1)
#> forcats * 0.5.0 2020-03-01 [1] CRAN (R 3.6.2)
#> foreach * 1.4.8 2020-02-09 [1] CRAN (R 3.6.2)
#> foreign * 0.8-76 2020-03-03 [4] CRAN (R 3.6.2)
#> formatR 1.7 2019-06-11 [1] CRAN (R 3.6.0)
#> Formula 1.2-3 2018-05-03 [1] CRAN (R 3.6.0)
#> fs 1.3.2 2020-03-05 [1] CRAN (R 3.6.2)
#> FSelector 0.31 2018-05-16 [1] CRAN (R 3.6.1)
#> furrr 0.1.0 2018-05-16 [1] CRAN (R 3.6.1)
#> future 1.16.0 2020-01-16 [1] CRAN (R 3.6.1)
#> gdata 2.18.0 2017-06-06 [1] CRAN (R 3.6.0)
#> generics 0.0.2 2018-11-29 [1] CRAN (R 3.6.0)
#> ggbiplot * 0.55 2019-08-26 [1] Github (vqv/ggbiplot@7325e88)
#> ggplot2 * 3.3.0 2020-03-05 [1] CRAN (R 3.6.2)
#> ggrepel * 0.8.2 2020-03-08 [1] CRAN (R 3.6.2)
#> gld * 2.6.2 2020-01-08 [1] CRAN (R 3.6.1)
#> globals 0.12.5 2019-12-07 [1] CRAN (R 3.6.1)
#> glue 1.3.2 2020-03-12 [1] CRAN (R 3.6.2)
#> gower 0.2.1 2019-05-14 [1] CRAN (R 3.6.0)
#> gplots * 3.0.3 2020-02-25 [1] CRAN (R 3.6.2)
#> gridExtra * 2.3 2017-09-09 [1] CRAN (R 3.6.2)
#> gtable 0.3.0 2019-03-25 [1] CRAN (R 3.6.0)
#> gtools * 3.8.1 2018-06-26 [1] CRAN (R 3.6.0)
#> haven 2.2.0 2019-11-08 [1] CRAN (R 3.6.1)
#> highr 0.8 2019-03-20 [1] CRAN (R 3.6.0)
#> hms 0.5.3 2020-01-08 [1] CRAN (R 3.6.1)
#> htmltools 0.4.0 2019-10-04 [1] CRAN (R 3.6.1)
#> htmlwidgets 1.5.1 2019-10-08 [1] CRAN (R 3.6.1)
#> httpuv 1.5.2 2019-09-11 [1] CRAN (R 3.6.1)
#> httr 1.4.1 2019-08-05 [1] CRAN (R 3.6.1)
#> igraph 1.2.4.2 2019-11-27 [1] CRAN (R 3.6.1)
#> inum 1.0-1 2019-04-25 [1] CRAN (R 3.6.0)
#> ipred 0.9-9 2019-04-28 [1] CRAN (R 3.6.0)
#> iterators * 1.0.12 2019-07-26 [1] CRAN (R 3.6.1)
#> jsonlite 1.6.1 2020-02-02 [1] CRAN (R 3.6.2)
#> kableExtra * 1.1.0 2019-03-16 [1] CRAN (R 3.6.0)
#> KernSmooth 2.23-16 2019-10-15 [4] CRAN (R 3.6.1)
#> knitr * 1.28 2020-02-06 [1] CRAN (R 3.6.2)
#> labeling 0.3 2014-08-23 [1] CRAN (R 3.6.0)
#> later 1.0.0 2019-10-04 [1] CRAN (R 3.6.1)
#> lattice * 0.20-40 2020-02-19 [4] CRAN (R 3.6.2)
#> lava 1.6.7 2020-03-05 [1] CRAN (R 3.6.2)
#> lavaan * 0.6-5 2019-08-28 [1] CRAN (R 3.6.1)
#> lazyeval 0.2.2 2019-03-15 [1] CRAN (R 3.6.0)
#> libcoin 1.0-5 2019-08-27 [1] CRAN (R 3.6.1)
#> lifecycle 0.2.0 2020-03-06 [1] CRAN (R 3.6.2)
#> limma * 3.40.6 2019-07-26 [1] Bioconductor
#> listenv 0.8.0 2019-12-05 [1] CRAN (R 3.6.1)
#> lmom 2.8 2019-03-12 [1] CRAN (R 3.6.1)
#> locfit 1.5-9.1 2013-04-20 [1] CRAN (R 3.6.0)
#> lubridate 1.7.4 2018-04-11 [1] CRAN (R 3.6.0)
#> magrittr 1.5 2014-11-22 [1] CRAN (R 3.6.0)
#> manipulateWidget 0.10.1 2020-02-24 [1] CRAN (R 3.6.2)
#> MASS * 7.3-51.5 2019-12-20 [1] CRAN (R 3.6.2)
#> MatchIt * 3.0.2 2018-01-09 [1] CRAN (R 3.6.2)
#> Matrix 1.2-18 2019-11-27 [4] CRAN (R 3.6.1)
#> MatrixModels 0.4-1 2015-08-22 [1] CRAN (R 3.6.0)
#> mboost 2.9-2 2020-02-18 [1] CRAN (R 3.6.2)
#> memoise 1.1.0 2017-04-21 [1] CRAN (R 3.6.0)
#> mice * 3.8.0 2020-02-21 [1] CRAN (R 3.6.2)
#> mime 0.9 2020-02-04 [1] CRAN (R 3.6.2)
#> miniUI 0.1.1.1 2018-05-18 [1] CRAN (R 3.6.0)
#> miRNAselector * 0.1.0 2020-03-18 [1] local
#> mnormt 1.5-6 2020-02-03 [1] CRAN (R 3.6.2)
#> ModelMetrics 1.2.2.2 2020-03-17 [1] CRAN (R 3.6.2)
#> modelr 0.1.6 2020-02-22 [1] CRAN (R 3.6.2)
#> mRMRe 2.1.0 2020-01-10 [1] CRAN (R 3.6.1)
#> munsell 0.5.0 2018-06-12 [1] CRAN (R 3.6.0)
#> mvtnorm * 1.1-0 2020-02-24 [1] CRAN (R 3.6.2)
#> nlme 3.1-145 2020-03-04 [4] CRAN (R 3.6.2)
#> nnet * 7.3-13 2020-02-25 [4] CRAN (R 3.6.2)
#> nnls 1.4 2012-03-19 [1] CRAN (R 3.6.0)
#> np 0.60-10 2020-02-06 [1] CRAN (R 3.6.2)
#> PairedData * 1.1.1 2018-06-02 [1] CRAN (R 3.6.2)
#> pamr 1.56.1 2019-04-22 [1] CRAN (R 3.6.1)
#> partykit 1.2-7 2020-03-06 [1] CRAN (R 3.6.2)
#> pbivnorm 0.6.0 2015-01-23 [1] CRAN (R 3.6.1)
#> pillar 1.4.3 2019-12-20 [1] CRAN (R 3.6.1)
#> pkgbuild 1.0.6 2019-10-09 [1] CRAN (R 3.6.1)
#> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 3.6.1)
#> pkgload 1.0.2 2018-10-29 [1] CRAN (R 3.6.0)
#> plotly * 4.9.2 2020-02-12 [1] CRAN (R 3.6.2)
#> plyr * 1.8.6 2020-03-03 [1] CRAN (R 3.6.2)
#> prettyunits 1.1.1 2020-01-24 [1] CRAN (R 3.6.2)
#> pROC * 1.16.1 2020-01-14 [1] CRAN (R 3.6.2)
#> processx 3.4.2 2020-02-09 [1] CRAN (R 3.6.2)
#> prodlim 2019.11.13 2019-11-17 [1] CRAN (R 3.6.1)
#> profileR * 0.3-5 2018-04-19 [1] CRAN (R 3.6.2)
#> promises 1.1.0 2019-10-04 [1] CRAN (R 3.6.1)
#> ps 1.3.2 2020-02-13 [1] CRAN (R 3.6.2)
#> psych 1.9.12.31 2020-01-08 [1] CRAN (R 3.6.2)
#> purrr * 0.3.3 2019-10-18 [1] CRAN (R 3.6.1)
#> quadprog 1.5-8 2019-11-20 [1] CRAN (R 3.6.1)
#> quantmod 0.4-16 2020-03-08 [1] CRAN (R 3.6.2)
#> quantreg 5.54 2019-12-13 [1] CRAN (R 3.6.1)
#> R.methodsS3 1.8.0 2020-02-14 [1] CRAN (R 3.6.2)
#> R.oo 1.23.0 2019-11-03 [1] CRAN (R 3.6.1)
#> R.utils 2.9.2 2019-12-08 [1] CRAN (R 3.6.2)
#> R6 2.4.1 2019-11-12 [1] CRAN (R 3.6.1)
#> randomForest 4.6-14 2018-03-25 [1] CRAN (R 3.6.2)
#> RColorBrewer * 1.1-2 2014-12-07 [1] CRAN (R 3.6.0)
#> Rcpp * 1.0.4 2020-03-17 [1] CRAN (R 3.6.2)
#> readr * 1.3.1 2018-12-21 [1] CRAN (R 3.6.1)
#> readxl 1.3.1 2019-03-13 [1] CRAN (R 3.6.0)
#> recipes 0.1.9 2020-01-07 [1] CRAN (R 3.6.1)
#> remotes 2.1.1 2020-02-15 [1] CRAN (R 3.6.2)
#> reprex 0.3.0 2019-05-16 [1] CRAN (R 3.6.1)
#> reshape * 0.8.8 2018-10-23 [1] CRAN (R 3.6.0)
#> reshape2 1.4.3 2017-12-11 [1] CRAN (R 3.6.0)
#> rgl 0.100.50 2020-02-21 [1] CRAN (R 3.6.2)
#> rJava 0.9-11 2019-03-29 [1] CRAN (R 3.6.0)
#> rlang 0.4.5 2020-03-01 [1] CRAN (R 3.6.2)
#> rmarkdown * 2.1 2020-01-20 [1] CRAN (R 3.6.2)
#> ROCR 1.0-7 2015-03-26 [1] CRAN (R 3.6.1)
#> ROSE * 0.0-3 2014-07-15 [1] CRAN (R 3.6.2)
#> rpart 4.1-15 2019-04-12 [1] CRAN (R 3.6.2)
#> rprojroot 1.3-2 2018-01-03 [1] CRAN (R 3.6.0)
#> rsq * 1.1 2018-09-27 [1] CRAN (R 3.6.2)
#> rstudioapi 0.11 2020-02-07 [1] CRAN (R 3.6.2)
#> rvest 0.3.5 2019-11-08 [1] CRAN (R 3.6.1)
#> RWeka 0.4-42 2020-02-02 [1] CRAN (R 3.6.2)
#> RWekajars 3.9.3-2 2019-10-19 [1] CRAN (R 3.6.1)
#> scales * 1.1.0 2019-11-18 [1] CRAN (R 3.6.1)
#> sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 3.6.0)
#> shiny 1.4.0.2 2020-03-13 [1] CRAN (R 3.6.2)
#> SparseM 1.78 2019-12-13 [1] CRAN (R 3.6.1)
#> stabs 0.6-3 2017-07-19 [1] CRAN (R 3.6.0)
#> stringi 1.4.6 2020-02-17 [1] CRAN (R 3.6.2)
#> stringr * 1.4.0 2019-02-10 [1] CRAN (R 3.6.2)
#> survival * 3.1-11 2020-03-07 [1] CRAN (R 3.6.2)
#> testthat 2.3.2 2020-03-02 [1] CRAN (R 3.6.2)
#> tibble * 2.1.3 2019-06-06 [1] CRAN (R 3.6.2)
#> tidyr * 1.0.2 2020-01-24 [1] CRAN (R 3.6.2)
#> tidyselect 1.0.0 2020-01-27 [1] CRAN (R 3.6.2)
#> tidyverse * 1.3.0 2019-11-21 [1] CRAN (R 3.6.2)
#> timeDate 3043.102 2018-02-21 [1] CRAN (R 3.6.0)
#> TTR 0.23-6 2019-12-15 [1] CRAN (R 3.6.1)
#> usethis * 1.5.1 2019-07-04 [1] CRAN (R 3.6.0)
#> vctrs 0.2.4 2020-03-10 [1] CRAN (R 3.6.2)
#> viridisLite 0.3.0 2018-02-01 [1] CRAN (R 3.6.0)
#> webshot 0.5.2 2019-11-22 [1] CRAN (R 3.6.1)
#> withr 2.1.2 2018-03-15 [1] CRAN (R 3.6.0)
#> xfun 0.12 2020-01-13 [1] CRAN (R 3.6.1)
#> xml2 1.2.5 2020-03-11 [1] CRAN (R 3.6.2)
#> xtable 1.8-4 2019-04-21 [1] CRAN (R 3.6.0)
#> xts 0.12-0 2020-01-19 [1] CRAN (R 3.6.2)
#> yaml 2.2.1 2020-02-01 [1] CRAN (R 3.6.2)
#> zoo 1.8-7 2020-01-10 [1] CRAN (R 3.6.1)
#>
#> [1] /home/konrad/R/x86_64-pc-linux-gnu-suppressMessages(library/3.6)
#> [2] /usr/local/lib/R/site-suppressMessages(library)
#> [3] /usr/lib/R/site-suppressMessages(library)
#> [4] /usr/lib/R/suppressMessages(libraryTo render this tutorial we used: